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Highlights 

Highlights  of  GAO-09-849,  a  report  to 
Congressional  Requesters 


HOMELAND  DEFENSE 

U.S.  Northern  Command  Has  a  Strong  Exercise 
Program,  but  Involvement  of  Interagency  Partners 
and  States  Can  Be  Improved 


Why  GAO  Did  This  Study 

U.S.  Northern  Command 
(NORTHCOM)  exercises  to  test 
preparedness  to  perform  its 
homeland  defense  and  civil  support 
missions.  GAO  was  asked  to  assess 
the  extent  to  which  NORTHCOM  is 
(1)  consistent  with  Department  of 
Defense  (DOD)  training  and 
exercise  requirements,  (2) 
involving  interagency  partners  and 
states  in  its  exercises,  (3)  using 
lessons  learned  and  corrective 
actions  to  improve  preparedness, 
and  (4)  integrating  its  exercises 
with  the  National  Exercise 
Program  (NEP).  To  do  this,  GAO 
reviewed  NORTHCOM  and  NEP 
guidance  and  postexercise 
documentation,  assessed 
NORTHCOM  compliance,  and 
compared  DOD  and  NEP  exercise 
requirements. 


What  GAO  Recommends 


GAO  is  making  recommendations 
to  DOD  to  direct  NORTHCOM  to 
consistently  involve  the  states  in 
planning,  executing,  and  assessing 
exercises  and  improve  oversight  of 
corrective  actions.  GAO  is  also 
recommending  that  DOD  define 
when  NORTHCOM  should  use  NEP 
planning  and  documentation 
requirements.  DOD  agreed  or 
partially  agreed  with  the 
recommendations  and  cited 
ongoing  and  future  efforts  to 
satisfy  the  recommendations’ 
intent.  DOD  did  not  fully  address  a 
recommendation  on  training  to 
NORTHCOM  staff  on  specific  state 
emergency  management  structures. 
GAO  believes  such  training  would 
benefit  NORTHCOM  personnel  in 
advance  of  a  crisis  and  for  exercise 
planning. 

View  GAO-09-849  or  key  components. 

For  more  information,  contact  Davi  M. 
D'Agostino,  (202)  512-5431, 
dagostinod@gao.gov. 


What  GAO  Found 

NORTHCOM’s  exercise  program  is  generally  consistent  with  the  requirements 
of  DOD’s  Joint  Training  System,  but  its  exercise  reporting  is  inconsistent. 

Since  the  command  was  established  in  2002,  NORTHCOM  has  conducted  13 
large-scale  exercises  and  generally  completed  exercise  summary  reports 
within  the  required  time  frame.  However,  those  reports  did  not  consistently 
include  certain  information,  such  as  areas  needing  improvement,  because 
NORTHCOM  lacks  guidance  that  specifies  exercise  reports’  content  and 
format,  potentially  impacting  its  ability  to  meet  internal  standards  for  planning 
and  execution  of  joint  exercises,  and  to  compare  and  share  exercise  results 
over  time  with  interagency  partners  and  states. 

Nineteen  federal  agencies  and  organizations  and  17  states  and  the  District  of 
Columbia  have  participated  in  one  or  more  of  the  seven  large-scale  exercises 
that  NORTHCOM  has  conducted  since  September  2005.  However, 

NORTHCOM  faces  challenges  in  involving  states  in  the  planning,  conduct,  and 
assessment  of  its  exercises,  such  as  adapting  its  exercise  system  and  practices 
to  involve  other  federal,  state,  local,  and  tribal  agencies  that  do  not  have  the 
same  practices  or  level  of  planning  resources.  Inconsistencies  with  how 
NORTHCOM  involves  states  in  exercises  are  occurring  in  part  because 
NORTHCOM  officials  lack  experience  dealing  with  states  and  do  not  have  a 
consistent  process  for  including  states  in  exercises.  Without  such  a  process, 
NORTHCOM  increases  the  risk  that  its  exercises  will  not  provide  benefits  for 
all  participants,  impact  the  seamless  exercise  of  all  levels  of  government,  and 
potentially  affect  NORTHCOM’s  ability  to  provide  civil  support  capabilities. 

NORTHCOM  has  a  systematic  lessons  learned  and  corrective  action  program 
to  improve  preparedness,  but  gaps  remain  with  collecting  and  sharing  lessons 
with  agency  and  state  partners  and  managing  corrective  actions.  Access  to  the 
system  NORTHCOM  uses  for  managing  exercise  observations  is  limited  for 
non-DOD  participants,  and  DOD  believes  that  the  Department  of  Homeland 
Security’s  system  is  not  adequately  protected  from  unauthorized  users. 
NORTHCOM’s  mitigation  steps  have  not  resolved  the  issues.  In  addition, 
about  20  percent  of  the  corrective  actions  tracked  by  NORTHCOM  were  being 
closed  prematurely  due  to  gaps  in  oversight.  Closing  issues  prematurely 
increases  the  risk  that  issues  will  reoccur  and  limits  the  knowledge  gained  and 
value  of  the  exercise. 

NORTHCOM  has  taken  steps  to  integrate  its  exercises  with  the  NEP,  but 
guidance  is  not  consistently  applied.  NORTHCOM  has  participated  in  several 
NEP  exercises  and  is  leading  its  first  major  NEP  exercise  in  the  fall  of  2009. 
However,  NORTHCOM  has  used  DOD’s  Joint  Training  System  planning  and 
documentation  requirements  rather  than  DHS’s  requirements,  because  NEP 
guidance  is  not  clear  on  what  exercise  planning  standard  should  be  used  and 
DOD  guidance  does  not  address  the  issue.  The  states  we  visited  use  NEP 
guidance.  Differences  between  NEP  and  DOD  guidance  could  affect  the  ability 
of  all  participants  to  develop  effective  working  relationships. 
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September  9,  2009 
Congressional  Requesters 

The  U.S.  homeland  continues  to  face  complex  and  dynamic  threats  and 
vulnerabilities  from  terrorism  and  other  catastrophic  incidents,  including 
natural  disasters,  industrial  accidents,  infrastructure  failures,  and 
infectious  diseases. 1  These  incidents  can  produce  devastating 
consequences  as  observed  during  the  September  11,  2001,  terrorist  attacks 
and  in  the  aftermath  of  Hurricane  Katrina,  which  required  an  effective  and 
coordinated  national  effort,  with  shared  goals  and  responsibilities  for 
protecting  and  defending  the  homeland.  The  complexity  of  national-level 
coordination  makes  preparing  for  an  incident  all  the  more  crucial.  2 
According  the  Quadrennial  Defense  Review,  unified  interagency  efforts 
among  the  Department  of  Defense  (DOD),  Department  of  Homeland 
Security  (DHS),  and  other  federal,  state,  and  local  agencies  are  required  to 
address  threats  to  the  homeland.3  Exercises  play  an  instrumental  role  in 
preparing  these  agencies  to  respond  to  an  incident  by  providing 
opportunities  to  test  plans,  improve  proficiency,  assess  capabilities  and 
readiness,  and  clarify  roles  and  responsibilities.4  Short  of  performance  in 
actual  operations,  exercises  provide  the  best  means  to  assess  the 
effectiveness  of  organizations  in  achieving  mission  preparedness. 
Exercises  also  provide  an  ideal  opportunity  to  enhance  preparedness  by 
collecting,  developing,  implementing,  and  disseminating  lessons  learned 
and  verifying  corrective  action  taken  to  resolve  previously  identified 
issues.6 


1Homeland  Security  Council,  National  Strategy  for'  Homeland  Security  (Washington,  D.C: 
October  2007),  1,  9,  13. 

2GAO,  National  Preparedness:  FEMA  Has  Made  Progress,  but  Needs  to  Complete  and 
Integrate  Planning,  Exercise,  and  Assessment  Efforts,  GAO-09-369  (Washington,  D.C.: 
Apr.  30,  2009). 

3DOD,  Quadrennial  Defense  Review  Report  (Washington,  D.C.:  Feb.  6,  2006),  87. 

4An  exercise  is  a  training  event  where  scenarios  are  enacted  to  assess  training  proficiency 
and  identify  lessons  learned  to  improve  preparedness.  Exercises  range  from  tabletop 
computer  simulations  to  the  full-scale  deployment  of  personnel  and  equipment. 

’U.S.  Northern  Command,  Commander’s  Training  Guidance  for  FY09-10  (Colorado 
Springs,  Colo.:  Apr.  8,  2008),  A-20. 
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Following  the  September  11  terrorist  attacks,  DOD  realized  the  need  for  a 
more  integrated  civilian  and  military  response  capability  for  any  future 
attack  on  the  homeland.  In  response,  DOD  established  the  U.S.  Northern 
Command  (NORTHCOM)  to  provide  and  manage  homeland  defense  and 
civil  support.  These  two  activities  are  among  DOD’s  contribution  to 
homeland  security.  DHS  is  the  lead  federal  agency  for  homeland  security, 
which  is  a  national  effort  to  prevent  terrorist  attacks  within  the  United 
States,  reduce  America’s  vulnerability  to  terrorism,  and  minimize  the 
damage  and  recover  from  attacks  that  do  occur.6  DOD  is  the  lead  federal 
agency  for  homeland  defense — defined  as  the  protection  of  U.S. 
sovereignty,  territory,  domestic  population,  and  critical  defense 
infrastructure  against  external  threats  and  aggression  against  the  United 
States.7  Civil  support  is  defense  support  of  civil  authorities — such  as  DHS 
or  another  federal  agency  or  state — for  domestic  emergencies  and  for 
designated  law  enforcement  and  other  activities.8  NORTHCOM  differs 
from  other  combatant  commands  in  that,  in  addition  to  Canada  and 
Mexico,  its  area  of  responsibility  includes  the  48  contiguous  states,  Alaska, 
the  District  of  Columbia,  Puerto  Rico,  and  the  U.S.  Virgin  Islands. 

Numerous  local,  state,  tribal,  and  federal  agencies  and  organizations — 
including  DOD,  DHS,  and  the  Federal  Emergency  Management  Agency 
(FEMA) — have  jurisdiction  over  or  can  coordinate  resources  within  the 
homeland  and,  therefore,  may  be  involved  in  the  response  to  an  incident. 
According  to  the  National  Defense  Strategy,  a  whole-of-govemment 
approach  is  only  possible  when  every  government  department  and  agency 
understands  the  core  competencies,  roles,  missions,  and  capabilities  of  its 
partners  and  works  together  to  achieve  common  goals.9  Ensuring  an 
effective  response  to  an  incident  will  require  that  federal  departments  and 
agencies,  states,  and  local  governments  conduct  integrated  disaster 
response  planning  and  test  these  plans  by  exercising  together.  10  To 
achieve  its  goal  of  being  ready  to  execute  joint  operations  and  ensure  a 
seamless  operating  environment,  NORTHCOM’s  training  goals  include 
maximizing  exercise  participation  with  federal,  state,  and  local  agencies 


6DOD,  Strategy  for  Homeland  Defense  and  Civil  Support,  5. 

'DOD,  Strategy  for  Homeland  Defense  and  Civil  Support,  5. 

sDOD,  Strategy  for  Homeland  Defense  and  Civil  Support,  5-6. 

9DOD,  National  Defense  Strategy  (Washington,  D.C.:  June  2008),  17-18. 

10Defense  Science  Board,  Unconventional  Operational  Concepts  and  the  Homeland 
(Washington,  D.C.:  March  2009),  37,  48,  50. 
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and  National  Guard  units.11  NORTHCOM  uses  long-established  DOD 
practices  for  training  and  exercises  to  test  and  enhance  the  preparedness 
of  its  personnel  and  forces  to  perform  its  homeland  defense  and  civil 
support  missions.  As  a  result,  DOD  has  more  experience  than  other  federal 
agencies  in  terms  of  the  scope  and  range  of  its  exercises  and  the 
robustness  of  the  process  for  conducting  them.  DOD’s  goal  is  to  improve 
the  homeland  defense  and  consequence  management  capabilities  of  its 
interagency  partners  by  leveraging  this  comparative  advantage  through 
shared  training  and  exercises. 12 

The  National  Exercise  Program  (NEP)  was  established  in  April  2007  under 
the  leadership  of  the  Secretary  of  Homeland  Security  to  prioritize  and 
coordinate  the  various  federal,  regional,  and  state  exercise  activities  and 
serves  as  the  principal  mechanism  for  examining  the  preparation  and 
efficiency  of  the  federal  government  to  respond  to  an  incident.13  The  NEP 
includes  a  series  of  national  exercises  projected  on  a  5-year  exercise 
schedule  and  has  established  a  four-tier  system  to  determine  the  relative 
priority  of  interagency  participation  in  each  exercise.  DOD  and  other 
federal  agencies,  states,  and  local  organizations  participate  in  these 
national-level  exercises.  FEMA  administers  the  NEP  and  has  developed 
Homeland  Security  Exercise  and  Evaluation  Program  guidance  to  provide 
standardized  policy,  methodology,  and  terminology  for  exercise  design, 
development,  conduct,  evaluation,  and  improvement  planning. 

In  our  previous  work  following  Hurricanes  Katrina  and  Rita,  we  identified 
a  need  to  improve,  among  other  things,  the  nation’s  disaster  response 
capabilities.  Overall,  capabilities  are  built  upon  the  appropriate 
combination  of  people,  skills,  processes,  and  assets.  Ensuring  that  needed 
capabilities  are  available  requires  effective  planning  and  coordination  in 
conjunction  with  training  and  exercises  in  which  the  capabilities  are 
realistically  tested  and  problems  identified  and  subsequently  addressed  in 
partnership  with  other  federal,  state,  and  local  stakeholders. 14  Specific  to 


1  'iJ.S.  Northern  Command,  Commander’s  Training  Guidance  for  FY09-10,  A-4. 

12DOD,  Quadrennial  Defense  Review  Report,  26-27. 

13DHS,  National  Exercise  Program  Implementation  Plan  (Washington  D.C.:  Apr.  11,  2007). 
DHS  issued  a  revised  Implementation  Plan  in  June  2008. 

14GAO,  Homeland  Security:  Observations  on  DHS  and  FEMA  Efforts  to  Prepare  for  and 
Respond  to  Major  and  Catastrophic  Disasters  and  Address  Related  Recommendations 
and  Legislation,  GAO-07-835T  (Washington,  D.C.:  May  16,  2007). 


Page  3 


GAO-09-849  Homeland  Defense 


DOD,  we  reported  that  prior  to  Hurricane  Katrina,  disaster  plans  and 
training  exercises  involving  DOD  were  insufficient,  and  did  not 
incorporate  lessons  learned  from  past  catastrophes  to  fully  delineate  the 
military  capabilities  that  could  be  needed  to  respond  to  a  catastrophic 
natural  disaster.  Moreover,  disaster  plans  had  not  been  tested  and  refined 
with  a  robust  exercise  program.  As  a  result  of  the  inadequate  plans — and 
the  lack  of  realistic  exercises  to  test  those  plans — a  lack  of  understanding 
existed  within  DOD  and  among  federal,  state,  and  local  responders  as  to 
the  types  of  assistance  and  capabilities  that  DOD  might  provide,  the  timing 
of  this  assistance,  and  the  respective  contributions  of  the  active-duty  and 
National  Guard  components.  We  recommended  that  DOD  establish 
milestones  and  expedite  the  development  of  detailed  plans  and  exercises 
to  fully  account  for  the  unique  capabilities  and  support  that  the  military  is 
likely  to  provide  to  civil  authorities  in  response  to  an  incident. 15  In 
addition,  we  recently  found  that  although  FEMA  has  developed  plans, 
guidance,  and  systems  to  design  exercises,  gather  lessons  learned,  and 
track  corrective  actions  to  implement  the  NEP,  it  faces  challenges  in 
ensuring  that  the  exercises  are  carried  out  in  accordance  with  Homeland 
Security  Exercise  and  Evaluation  Program  guidance.  We  recommended, 
among  other  things,  that  FEMA  establish  a  program  management  plan  and 
better  ensure  that  exercises  follow  Homeland  Security  Exercise 
Evaluation  Program  guidance,  for  example,  by  revising  its  grant 
monitoring  guidance  to  include  a  checklist  of  specific  requirements  for 
states  that  receive  grant  funds.18 

In  order  to  better  understand  NORTHCOM’s  efforts  to  effectively  prepare 
for  its  homeland  defense  and  civil  support  missions,  we  were  asked  to 
review  NORTHCOM’s  exercise  program.  This  review  is  the  final  part  of  a 
broader  congressional  request  to  review  NORTHCOM’s  efforts  to  plan  and 
coordinate  its  homeland  defense  and  civil  support  missions.  We  have 
issued  two  previous  reports  in  support  of  that  request.17  For  this  report, 
our  objectives  were  to  determine  the  extent  to  which  (1)  NORTHCOM’s 


15GAO,  Hurricane  Katrina:  Better  Plans  and  Exercises  Needed  to  Guide  the  Military’s 
Response  to  Catastrophic  Natural  Disasters,  GAO-06-643  (Washington,  D.C.:  MaylG,  2006). 

16GAO-09-369. 

17GAO,  Homeland  Defense:  U.S.  Northern  Command  Has  Made  Progress  but  Needs  to 
Address  Force  Allocation,  Readiness,  Tracking  Gaps,  and  Other  Issues,  GAO-08-251 
(Washington,  D.C.:  Apr.  16,  2008);  and  Homeland  Defense:  Steps  Have  Been  Taken  to 
Improve  U.S.  Northern  Command’s  Coordination  with  the  States  and  the  National 
Guard  Bureau,  but  Gaps  Remain,  GAO-08-252  (Washington,  D.C.:  Apr.  18,  2008). 
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exercise  program  is  consistent  with  DOD  training  and  exercise 
requirements;  (2)  NORTHCOM  involves  relevant  interagency  partners  and 
states  in  planning,  conducting,  and  assessing  exercises;  (3)  NORTHCOM  is 
using  lessons  learned  and  corrective  actions  during  exercises  to  improve 
mission  preparedness;  and  (4)  NORTHCOM  is  integrating  its  exercises 
with  the  NEP. 

In  conducting  this  review,  we  generally  focused  our  scope  on 
NORTHCOM’s  large-scale  exercises18  conducted  since  Hurricane  Katrina 
made  landfall  in  August  2005.  To  determine  the  extent  to  which 
NORTHCOM’s  exercise  program  is  consistent  with  DOD  exercise 
requirements  and  includes  relevant  exercise  partners,  we  reviewed 
exercise  documentation  for  all  13  large-scale  exercises  the  command 
performed  since  it  was  established  in  2002;  evaluated  NORTHCOM’s 
compliance  with  requirements  established  in  DOD  guidance;  and  assessed 
the  level  of  interagency  and  state  participation  in  NORTHCOM’s  large- 
scale  exercises.  We  also  interviewed  Office  of  the  Secretary  of  Defense, 
Joint  Staff,  NORTHCOM,  FEMA,  FEMA  regional  offices,  and  state  officials 
with  knowledge  of  and  experience  with  NORTHCOM’s  exercise  program. 
We  selected  a  nongeneralizable  sampling  of  states  based  on  the  extent  to 
which  they  have  participated  in  major  NORTHCOM  exercises  since 
Hurricane  Katrina  and  the  varying  scenarios  of  the  exercises.  The  states 
we  selected  played  a  major  role  in  NORTHCOM  exercises  by  having  a 
portion  of  the  exercise  conducted  in  their  state  and  having  various  state 
agencies  and  officials  participate.  States  we  selected  include  Arizona, 
California,  Michigan,  Oregon,  Rhode  Island,  and  Washington.  We  also  met 
with  Nevada  officials  who  participated  in  a  NORTHCOM  exercise  prior  to 
Hurricane  Katrina — Determined  Promise  03 — to  provide  context  to  the 
extent  that  changes  may  have  been  made  to  NORTHCOM’s  exercise 
program  and  help  develop  our  state  selection  methodology.  To  determine 
the  extent  to  which  NORTHCOM  is  using  lessons  learned  during  exercises 
to  improve  mission  preparedness,  we  reviewed  DOD,  NORTHCOM,  and 
DHS’s  NEP  guidance  for  lessons  learned.  Based  on  this  guidance,  we 
assessed  the  management  of  all  unclassified  exercise  observations  and 
issues  identified  from  its  last  6  large-scale  exercises  by  determining  each 
record’s  status  (open  or  closed),  type  (issue  or  lesson  learned),  and 
disposition  after  NORTHCOM  staff  have  acted  on  these  records.  Finally,  to 


18Large-scale  exercises  train  and  evaluate  actual  forces  deployed  in  a  field  environment 
under  simulated  scenarios.  NORTHCOM  generally  conducts  two  large-scale  exercises 
annually. 
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determine  the  extent  to  which  NORTHCOM  is  integrating  its  exercises 
with  the  NEP  we  reviewed  DOD  and  DHS  guidance  regarding  the  NEP  and 
how  DOD  and  its  subordinate  commands  are  required  to  or  should 
participate.  We  conducted  our  review  between  June  2008  and  September 
2009  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives. 


Results  in  Brief 


NORTHCOM’s  exercise  program  is  generally  consistent  with  the  overall 
policies  and  practices  of  DOD’s  Joint  Training  System,  but  the  way  it 
reports  on  exercises  is  inconsistent.  DOD  and  NORTHCOM  guidance 
require  that  the  command  establish  an  exercise  program  consistent  with 
the  Joint  Training  System  and  complete  postexercise  documentation 
within  an  established  time  frame.19  NORTHCOM  has  developed  a 
comprehensive  exercise  program  consistent  with  DOD’s  Joint  Training 
System,  conducted  13  large-scale  exercises  since  it  was  created  in  2002, 
and  generally  completed  exercise  summary  reports  within  the  required 
time  frame.  However,  those  reports  did  not  consistently  include  certain 
information;  for  example,  only  5  of  the  11  exercise  summary  reports 
included  an  identified  section  on  lessons  learned.  DHS  currently  has  a 
template  for  exercise  documentation  that  includes  guidance  on  content 
and  format;  however,  NORTHCOM  does  not  follow  this  template  and 
neither  DOD  nor  NORTHCOM  has  similar  guidance  that  specifies  content 
and  format.  Without  a  consistent  record  of  what  occurred  during  an 
exercise,  NORTHCOM  cannot  ensure  that  it  has  met  internal  standards  for 
planning  and  execution  of  joint  exercises,  compare  exercise  results  over 
time,  and  share  lessons  learned  with  interagency  partners  and  states.  We 
are  recommending  that  NORTHCOM’s  Commander  establish  criteria  for 
the  format  and  content  of  postexercise  documentation. 

Interagency  partners  and  states  have  participated  in  NORTHCOM 
exercises,  but  NORTHCOM  faces  challenges  involving  interagency 
partners  and  states  in  planning,  conducting,  and  assessing  exercises. 


^Chairman  of  the  Joint  Chiefs  of  Staff,  Joint  Training  Manual  for  the  Armed  Forces  of  the 
United  States,  3500.03B  (Washington,  D.C.:  Aug.  31,  2007),  A-l;  U.S.  Northern  Command, 
Operations:  Exercise  Program,  Instruction  10-156  (Colorado  Springs,  Colo.:  June  2,  2008). 
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NORTHCOM  guidance  requires  that  NORTHCOM  maximize  exercise 
participation  with  interagency  partners  and  states  to  achieve  preparedness 
goals  and  ensure  a  seamless  operating  environment.20  Seventeen  federal 
agencies  and  organizations  and  17  states  and  the  District  of  Columbia  have 
participated  in  one  or  more  of  the  seven  large-scale  exercises  that 
NORTHCOM  has  conducted  since  September  2005.  We  found  that  the 
states  we  visited  derived  benefits  from  their  participation  in  these 
exercises.  For  example,  officials  from  3  states  told  us  this  was  their  first 
opportunity  to  interact  with  federal  military  forces.  However,  we  found 
that  challenges  remain  which  have  resulted  in  inconsistencies  in  the  way 
that  NORTHCOM  involves  the  states  in  its  exercises.  One  of  DOD’s 
challenges  is  adapting  its  exercise  system  and  practices  to  accommodate 
the  coordination  and  involvement  of  other  federal,  state,  local,  and  tribal 
agencies  that  do  not  have  the  same  kinds  of  practices  or  level  of  planning 
effort.  NORTHCOM  also  faces  the  challenge  of  balancing  its  training 
objectives  with  those  of  state  agencies  and  organizations,  particularly 
given  the  limited  resources  and  funding  states  have  available  to  exercise. 
While  state  and  local  governments  seek  to  exercise  their  first  responder 
capabilities  before  having  their  resources  overwhelmed  and  needing  to 
seek  federal  assistance,  NORTHCOM’s  goal  is  to  exercise  its  capability  to 
provide  support  to  civil  authorities  when  local,  state,  and  other  federal 
resources  are  overwhelmed.  As  a  result  of  this  challenge,  officials  from  5 
states  told  us  that  all  of  their  needs  were  not  fully  met  during  the 
exercises,  for  example,  due  to  large-scale,  unrealistic  scenarios  that 
overwhelmed  the  states’  resources  before  they  had  the  opportunity  to 
exercise  their  training  objectives.  Inconsistencies  with  how  NORTHCOM 
involves  states  in  planning,  conducting,  and  assessing  exercises  are 
occurring  in  part  because  NORTHCOM  officials  lack  experience  in  dealing 
with  the  differing  emergency  management  structures,  capabilities,  and 
needs  of  the  states.  Inconsistencies  are  also  occurring  because 
NORTHCOM  has  not  established  a  process  for  including  states  in 
exercises,  such  as  consistent  procedures  for  requesting  state  involvement 
in  exercises  through  DHS/FEMA  or  the  National  Guard  Bureau.  Without  an 
informed  and  consistent  process,  NORTHCOM  increases  the  risk  that  its 
exercises  will  not  provide  benefits  for  all  participants,  impacting  the 
seamless  exercise  of  all  levels  of  government  and  potentially  affecting 
NORTHCOM’s  ability  to  provide  support  to  civil  authorities.  We  are 
recommending  that  (1)  DOD  work  with  DHS  to  establish  guidance  and 
procedures  for  requesting  state  participation  and  involving  states  in 


20U.S.  Northern  Command,  Commander’s  Training  Guidance  FY  09-10,  A-4. 
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planning,  executing,  and  assessing  exercises  and  (2)  NORTHCOM’s 
Commander  develop  a  training  plan  for  NORTHCOM  staff  on  state 
emergency  management  structures  and  relevant  issues  related  to  working 
with  civilian  state  and  local  emergency  management  officials. 

NORTHCOM  has  a  systematic  lessons  learned  and  corrective  action 
program  to  improve  preparedness,  but  gaps  remain  with  collecting  and 
sharing  lessons  with  agency  and  state  partners  and  managing  corrective 
actions.  DOD  and  NORTHCOM  guidance  requires  that  NORTHCOM 
identify  lessons  learned  during  the  course  of  normal  operations,  exercises, 
and  real-world  events;  share  valid  observations  and  findings  as  widely  as 
possible;  and  track  them  until  the  corrective  actions  are  verified  during 
subsequent  events  or  exercises.21  NORTHCOM  uses  DOD’s  Joint  Lessons 
Learned  Information  System  (JLLIS),  an  automated  tool  for  managing  and 
tracking  observations,  and  a  corrective  action  process  to  manage  issues 
requiring  action  for  resolution.  Access  to  this  system  is  limited  for  non- 
DOD  participants,  and  DOD  believes  that  DHS’s  system  is  not  adequately 
protected  from  potential  unauthorized  users.  NORTHCOM  has  taken  steps 
to  address  these  issues,  such  as  placing  a  template  on  NORTHCOM’s 
restricted  access  portal  to  provide  a  venue  for  non-DOD  officials  to  record 
exercise  observations,  but  these  efforts  have  not  been  successful.  Because 
NORTHCOM  is  not  fully  involving  other  federal  agencies  and  states  in  its 
lessons  learned  process,  it  is  missing  opportunities  to  learn  lessons  from 
an  exercise.  For  example,  officials  from  two  states  did  not  provide 
NORTHCOM  with  observations  from  exercises  because  they  did  not 
attend  the  command’s  postexercise  reviews  for  varying  reasons,  such  as 
limited  resources.  As  a  result,  NORTHCOM  risks  the  reoccurrence  of  these 
issues.  We  also  found  that  about  20  percent  of  the  corrective  actions 
tracked  by  NORTHCOM  were  being  closed  prematurely  because,  under 
existing  procedures,  NORTHCOM  exercise  directorate  officials  do  not 
have  oversight  over  issues  that  are  resolved  within  other  directorates  or 
are  not  giving  long-standing  issues  the  sustained  management  attention  to 
ensure  resolution.  A  NORTHCOM  official  told  us  that  they  do  not  have  the 
staff  necessary  to  oversee  the  actions  on  records  handled  within  the  other 
directorates.  Closing  issues  requiring  corrective  actions  prematurely  could 
increase  the  risk  that  they  could  reoccur  and  limits  the  knowledge  gained 


21Chairman  of  the  Joint  Chiefs  of  Staff,  Joint  Lessons  Learned  Program. ,  Instruction 
3150.25D  (Washington,  D.C.:  Oct.  10,  2008),  A-l  and  A-6;  U.S.  Northern  Command, 
Commander’s  Training  Guidance  FY  09-10  ,  A-4,  A-8,  A-20,  A-2-8;  U.S.  Northern 
Command,  Lessons  Learned  Program  and  Com'ective  Action  Board  Process,  Instruction 
16-166  (Colorado  Springs,  Colo.:  Mar.  16,  2009),  2. 
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and  value  of  the  exercise.  We  are  recommending  that  (1)  NORTHCOM’s 
Commander  establish  and  publicize  procedures  for  non-DOD  exercise 
participants  to  submit  observations,  (2)  the  Chairman  of  the  Joint  Chiefs 
of  Staff  direct  the  Joint  Staff  to  work  with  DHS  to  either  resolve 
information  assurance  issues  so  that  combatant  commands  can  post 
Exercise  Summary  Reports  with  lessons  learned  on  DHS’s  Lessons 
Learned  Information  Sharing  system  or  establish  an  alternative  method  to 
systematically  collect  and  share  lessons  learned,  and  (3)  the  Chairman  of 
the  Joints  Chiefs  of  Staff  direct  the  Joint  Staff  to  revise  the  joint  lessons 
learned  operating  instruction  to  include  procedures  to  ensure  that 
corrective  actions  are  implemented  and  verified  in  a  subsequent  exercise 
or  operation  before  being  closed. 

NORTHCOM  has  taken  steps  to  integrate  its  exercises  with  the  NEP,  such 
as  participating  in  several  NEP  exercises,  including  combining  two  of  its 
large-scale  exercises  with  NEP  exercises.  However,  NORTHCOM  has  not 
consistently  applied  NEP  planning  and  documentation  requirements, 
because  NEP  guidance  is  unclear  on  the  extent  to  which  federal  agencies 
should  follow  NEP  requirements  and  DOD  guidance  does  not  specifically 
address  this  issue.  NEP  guidance  provides  policies  and  tools  for  the 
design,  planning,  conduct,  and  evaluation  of  exercises — known  as  the 
Homeland  Security  Exercise  and  Evaluation  Program.22  FEMA  requires 
that  entities,  such  as  states,  receiving  homeland  security  grant  funding  for 
their  exercises  use  Homeland  Security  Exercise  and  Evaluation  Program 
tools,  such  as  reporting  templates.  We  reviewed  NEP  guidance,  such  as  the 
Implementation  Plan,  and  found  it  does  not  clearly  state  the  extent  to 
which  federal  agencies  are  required  to  follow  the  Homeland  Security 
Exercise  and  Evaluation  Program  planning  and  documentation  guidance. 
As  a  result  of  this  unclear  guidance,  we  found  that  agency  officials  have 
varying  interpretations  of  the  requirements.  A  DOD  and  a  Joint  Staff 
official  told  us  that  NEP  guidance  does  not  require  agencies  to  use  the 
Homeland  Security  Exercise  Evaluation  Program  even  for  NEP  exercises; 
therefore,  NORTHCOM  uses  the  Joint  Training  System  as  the  basis  for 
planning,  conducting,  and  assessing  exercises.  FEMA  officials  told  us  that 
federal  agencies  should  use  the  Homeland  Security  Exercise  Evaluation 
Program  for  exercises  that  require  some  level  of  interagency  participation, 


22DHS,  National  Exercise  Program  Implementation  Plan  (Washington,  D.C.:  June  20, 
2008). 
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so  that  consistent  approaches  are  used  by  the  various  participants.23 
Neither  DOD  nor  NORTHCOM  guidance  specifically  addresses  this  issue.24 
We  recognize  that  DOD  and  NORTHCOM  must  meet  their  own  mission 
and  exercise  requirements  and  the  Joint  Training  System  may  be  best 
suited  for  NORTHCOM’s  exercises;  however,  all  of  the  states  we  visited 
use  Homeland  Security  Exercise  and  Evaluation  Program  guidance.  We 
found  that  having  differing  sets  of  guidance  makes  exercise  development 
more  difficult  and  potentially  limits  the  benefits  for  participating  states. 
Further,  inconsistent  approaches  to  the  development  and  content  of 
postexercise  documentation  may  affect  the  ability  of  organizations  to  fully 
learn  lessons  identified  in  exercises.25  We  believe  that  achieving  national 
preparedness  requires  a  whole-of-government  approach  and  is  a  shared 
responsibility  among  federal,  state,  and  local  governments  and 
organizations  and  an  integration  of  their  various  standards,  policies,  and 
procedures  into  the  national  system.26  In  the  absence  of  clear  NEP 
guidance  on  this  issue,  DOD  should  ensure  that  its  components  clearly 
understand  when  use  of  NEP  planning  and  reporting  guidelines  is 
appropriate.  We  are  recommending  that  the  Secretary  of  Defense  and  the 
Joint  Staff  provide  specific  guidance  that  defines  the  conditions  under 
which  the  combatant  commands  should  follow  the  Homeland  Security 
Exercise  and  Evaluation  Program  planning  and  documentation 
requirements  or  the  DOD’s  Joint  Training  System  should  be  modified  for 
those  civil  support  exercises. 

DOD  provided  written  comments  on  a  draft  of  this  report  and  provided 
technical  comments  which  we  incorporated  into  the  final  report  as 
appropriate.  DOD  agreed  or  partially  agreed  with  all  our  recommendations 
and  described  actions  it  is  taking  or  plans  to  take  to  implement  them.  DOD 


23FEMA  officials  stated  that  federal  departments  and  agencies  should  be  held  accountable 
for  meeting  key  Homeland  Security  Exercise  Evaluation  Program  requirements,  but  that 
FEMA  lacks  the  authority  to  ensure  compliance.  While  we  recognize  that  FEMA’s  role  is 
generally  to  coordinate,  guide,  and  support,  we  believe  that  FEMA’s  expanded  leadership 
role  under  the  Post-Katrina  Act  of  2006  provides  FEMA  opportunities  to  instill  a  shared 
sense  of  responsibility  and  accountability  on  the  part  of  all  agencies.  GAO-09-369.  The  Post- 
Katrina  Act  was  enacted  as  Title  VI  of  the  Department  of  Homeland  Security 
Appropriations  Act,  2007,  Pub.  L.  No.  109-295,  (2006). 

24NORTHCOM  guidance  states  the  Vigilant  Guard  Program  should  use  Homeland  Security 
Exercise  and  Evaluation  Program  formats  to  encourage  civilian  agency  participation.  U.S. 
Northern  Command,  Commander’s  Training  Guidance:  Fiscal  Year  2009-2010,  p.  A-18. 

25Defense  Science  Board,  Unconventional  Operational  Concepts  and  the  Homeland. 

26GAO-09-369. 
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generally  agreed  with  the  recommendation  to  work  with  DHS  to  establish 
guidance  and  procedures  for  requesting  state  participation  and  involving 
states  in  planning,  executing,  and  assessing  exercises.  DOD  agreed  that 
better  coordination  for  interfacing  with  state  officials  can  be  achieved. 
However,  DOD  noted  that  NORTHCOM  is  the  primary  training  audience 
for  its  exercises  and  that  the  recommendation  should  also  be  addressed  to 
an  interagency  exercise  planning  committee.  We  noted  NORTHCOM’s 
considerable  efforts  to  involve  state  and  interagency  exercise  partners. 
However,  we  believe  that  the  effective  involvement  of  and  interaction  with 
state  and  other  federal  partners  is  a  critical  component  of  improving  and 
maintaining  NORTHCOM’s  preparedness  for  providing  support  to  civil 
authorities  regardless  of  whether  it  is  a  strictly  NORTHCOM-sponsored 
exercise  or  an  event  conducted  under  the  National  Exercise  Program. 
Therefore,  developing  procedures  to  improve  coordination  with  the  states 
can  only  assist  DOD  and  all  its  interagency  and  state  partners.  We  also 
noted  that  the  interagency  exercise  planning  committee  is  one  venue  at 
which  DOD  can  effectively  coordinate  with  its  interagency  partners. 
Although  DOD  agreed  with  the  recommendation  on  a  training  plan  for 
NORTHCOM  staff  and  discussed  current  and  future  NORTHCOM  training 
efforts,  its  response  did  not  address  the  need  for  NORTHCOM  staff  to  be 
provided  more  thorough  training  on  specific  emergency  management 
organizations  and  structures  of  each  state.  We  continue  to  believe  that 
such  training  is  needed  since  it  is  an  important  element  of  preparedness 
for  exercises  and  for  actually  providing  civil  support  in  the  event  of  a 
major  incident.  A  summary  of  DOD’s  comments  and  a  summary  of  our 
response  to  these  comments  follow  the  Recommendations  for  Executive 
Action  section  of  this  report.  DOD’s  written  comments  are  reprinted  in 
appendix  III.  DHS  also  reviewed  a  draft  of  this  report  and  provided 
technical  comments,  which  we  incorporated  into  the  final  report  as 
appropriate. 


Background 


The  U.S.  homeland  continues  to  face  an  uncertain,  complex  security 
environment  with  the  potential  for  terrorist  incidents  and  natural  disasters 
which  can  produce  devastating  consequences.  Ensuring  an  effective 
response  will  require  that  federal  departments  and  agencies,  states,  and 
local  governments  conduct  integrated  disaster  response  planning  and  test 
these  plans  by  exercising  together. 


Training  and  Exercises  Exercises  play  an  instrumental  role  in  preparing  the  nation  to  respond  to 

an  incident  by  providing  opportunities  to  test  emergency  response  plans, 
evaluate  response  capabilities,  assess  the  clarity  of  established  roles  and 
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responsibilities,  and  improve  proficiency  in  a  simulated,  risk-free 
environment.27  Short  of  performance  in  actual  operations,  exercises 
provide  the  best  means  to  assess  the  effectiveness  of  organizations  in 
achieving  mission  preparedness.  Exercises  provide  an  ideal  opportunity  to 
collect,  develop,  implement,  and  disseminate  lessons  learned  and  to  verify 
corrective  action  taken  to  resolve  previously  identified  issues.28  Sharing 
positive  experiences  reinforces  positive  behaviors,  doctrine,  and  tactics, 
techniques,  and  procedures,  while  disseminating  negative  experiences 
highlights  potential  challenges  in  unique  situations  or  environments  or 
identifies  issues  that  need  to  be  resolved.29  According  to  the  National 
Response  Framework,  well-designed  exercises  improve  interagency 
coordination  and  communications,  highlight  capability  gaps,  and  identify 
opportunities  for  improvement.30  There  are  various  types  of  exercises 
ranging  from  tabletop  exercises  that  involve  key  personnel  discussing 
simulated  scenarios  in  informal  settings  to  a  full-scale  exercise,  including 
many  agencies,  jurisdictions,  and  disciplines  and  a  “boots  on  the  ground” 
response,  such  as  firefighters  decontaminating  mock  victims. 


Military  Mission  and 
Organization 


DOD  established  the  Office  of  the  Assistant  Secretary  of  Defense  for 
Homeland  Defense  and  Americas’  Security  Affairs  to  oversee  homeland 
defense  activities  for  DOD,  under  the  authority  of  the  Under  Secretary  of 
Defense  for  Policy,  and,  as  appropriate,  in  coordination  with  the  Chairman 
of  the  Joint  Chiefs  of  Staff.  This  office  develops  policies,  conducts 
analysis,  provides  advice,  and  makes  recommendations  on  homeland 
defense,  defense  support  of  civil  authorities,  emergency  preparedness,  and 
domestic  crises  management  matters  within  the  department.  The  assistant 
secretary  assists  the  Secretary  of  Defense  in  providing  policy  directions  to 
NORTHCOM  and  other  applicable  combatant  commands  to  guide  the 


2iA  training  event  focuses  primarily  on  improving  individual  or  collective  ability  to  perform, 
such  as  academic  or  field  training.  An  exercise  focuses  primarily  on  evaluating  capability 
or  an  element  of  capability,  such  as  a  plan  or  policy.  Necessary  training  should  take  place 
prior  to  an  exercise. 

28U.S.  Northern  Command,  Commander’s  Training  Guidance  for  FY09-10,  A-20. 

29U.S.  Northern  Command,  Lessons  Learned  Program,  and  Corrective  Action  Board 
Process,  Instruction  16-166  (Colorado  Springs,  Colo.:  Mar.  16,  2009),  2. 

30  The  framework  is  a  guide  to  how  the  nation  conducts  all-hazards  response  and  is 
intended  to  capture  specific  authorities  and  best  practices  for  managing  incidents  that 
range  from  the  serious  but  purely  local,  to  large-scale  terrorist  attacks  or  catastrophic 
natural  disasters.  DHS,  National  Response  Framework  (Washington,  D.C.:  January  2008). 
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development  and  execution  of  homeland  defense  plans  and  activities.  This 
direction  is  provided  through  the  Chairman  of  the  Joint  Chiefs  of  Staff. 

The  Chairman  of  the  Joint  Chiefs  of  Staff,  as  principal  military  advisor  to 
the  President  and  Secretary  of  Defense,  has  numerous  responsibilities 
relating  to  homeland  defense  and  civil  support,  including  providing  advice 
on  operational  policies,  responsibilities,  and  programs.  Furthermore,  the 
Chairman  of  the  Joint  Chiefs  of  Staff  and  the  Joint  Staff  are  responsible  for 
formulating  joint  training  policy  and  doctrine.31  The  Joint  Staff  assists  the 
Chairman  by  facilitating  implementation  of  the  Chairman’s  joint  training 
programs,  including  the  Joint  Training  System,  Chairman’s  sponsored 
exercise  program,  and  joint  exercise  program. 


NORTHCOM  NORTHCOM  is  the  military  command  responsible  for  planning,  organizing, 

and  executing  DOD’s  homeland  defense  and  civil  support  missions  within 
its  area  of  responsibility — the  continental  United  States  (including  Alaska) 
and  territorial  waters  (see  fig.  I).32  Homeland  defense  is  the  protection  of 
U.S  territory,  sovereignty,  domestic  population,  and  critical  defense 
infrastructure  against  external  threats  and  aggression.  DOD  is  the  primary 
federal  agency  responsible  for  homeland  defense  operations,  such  as  air 
defense,  and  NORTHCOM  is  the  combatant  command  responsible  for 
commanding  and  coordinating  a  response  to  a  homeland  defense 
incident.38  To  carry  out  its  homeland  defense  mission,  NORTHCOM  is  to 
conduct  operations  to  deter,  prevent,  and  defeat  threats  and  aggression 
aimed  at  the  United  States. 


31Chairman  of  the  Joint  Chiefs  of  Staff,  Instruction  3500. 01E,  Joint  Training  Policy  and 
Guidance  for  the  Anned  Forces  of  the  United  States  (Washington,  D.C.:  May  31,  2008). 

32DOD,  Strategy  for  Homeland  Defense  and  Civil  Support,  8. 

33Homeland  defense  is  considered  DOD’s  portion  of  the  broader  area  of  homeland  security. 
DHS  is  the  lead  primary  federal  agency  for  homeland  security  issues,  which  is  a  concerted 
national  effort  to  prevent  terrorist  attacks  within  the  United  States,  reduce  America’s 
vulnerability  to  terrorism,  and  minimize  the  damage  and  recover  from  attacks  that  do 
occur.  Homeland  Security  Council,  National  Strategy  for  Homeland  Security,  3;  DOD, 
Strategy  for  Homeland  Defense  and  Civil  Support,  5. 
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Figure  1 :  NORTHCOM’s  Area  of  Responsibility 


Alaska  falls  within 
NORTHCOM’s  area 
of  responsibility,  but 
military  forces  in  the 
state  are  assigned  to 
U.S.  Pacific  Command 


Sources:  GAO  analysis;  Map  Resources  (map). 


NORTHCOM’s  second  mission  is  civil  support  or  defense  support  of  civil 
authorities.  Civil  support  is  DOD  support  to  U.S.  civilian  authorities,  such 
as  DHS,  for  domestic  emergencies,  both  natural  and  man-made,  and 
includes  the  use  of  DOD  personnel — federal  military  forces  and  DOD’s 
career  civilian  and  contractor  personnel — and  DOD  agency  and 
component  resources.  Because  these  missions  are  complex  and 
interrelated,  they  require  significant  interagency  coordination.  Civil 
support  missions  include  domestic  disaster  relief  operations  for  incidents 
such  as  fires,  hurricanes,  floods,  and  earthquakes.  Such  support  also 
includes  counterdrug  operations  and  management  of  the  consequences  of 
a  terrorist  incident  employing  a  weapon  of  mass  destruction.  DOD  is  not 
the  primary  federal  agency  for  such  missions  (unless  so  designated  by  the 
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President)  and  thus  provides  defense  support  of  civil  authorities  only 
when  (1)  state,  local,  and  other  federal  resources  are  overwhelmed  or 
unique  military  capabilities  are  required;  (2)  assistance  is  requested  by  the 
primary  federal  agency;  and  (3)  NORTHCOM  is  directed  to  do  so  by  the 
President  or  the  Secretary  of  Defense. 34  See  fig.  2  for  the  pathway  for 
requesting  DOD  and  NORTHCOM  assistance  during  an  incident. 


34DHS,  National  Response  Framework-,  Joint  Chiefs  of  Staff,  Joint  Pub.  3-28,  Civil  Support 
(Washington,  D.C.:  Sept.  14,  2007). 
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Figure  2:  Pathways  for  Requesting  Military  Assistance  for  Incident  Response 


President  of  the  United  States 

Declares  incident  as  either  a  major  disaster  or 
emergency  under  the  Stafford  Act 


Secretary  of  Homeland  Security/ 
FEMA  Administrator 


FEMA  Regional 
Administrator 


Governor  and  state  officials, 
including  emergency 
management  officials 


Local  elected  and 
emergency  management 
officials 


Incident  commander  and 
local  first  responders 


Joint  Field  Office  (Temporary) 

Established  to  coordinate  federal 
support  for  response,  but  does  not 
manage  on-scene  operations 


-► 


Secretary  of  Defense 

Approves  FEMA  requests  for 
assistance 


State 

Coordinating 

Officer 


Federal 

Coordinating 

Officer 


Other  federal 
departments 


NORTHCOM 


Joint  Task 
Force 

Commander 

(Temporary) 


Defense 

Coordinating 

Officer 


Designated  forces  support 
as  directed 


Request  for  federal  assistance 


Request  for  federal  assistance  after  Stafford 

declaration 

Approves  assistance 

Appoints  Federal  Coordinating  Officer 

Requests  assistance  directly  to  DOD 

Directs  assistance  (when  assigned) 

Delivers  assistance 


Source:  GAO  analysis  of  DOD  and  DHS  documentation. 
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NORTHCOM  conducts  or  participates  in  exercises  to  improve  readiness  to 
perform  its  assigned  missions.  The  command  annually  conducts  2  large- 
scale  exercises35 — Ardent  Sentry  and  Vigilant  Shield — and  participates  in 
over  30  smaller  command,  regional,  state,  and  local  exercises.  Each  Ardent 
Sentry  and  Vigilant  Shield  training  event  emphasizes  one  of  the  key 
missions  while  at  the  same  time  including  elements  of  the  other.  Ardent 
Sentry  emphasizes  the  civil  support  missions;  Vigilant  Shield  the  homeland 
defense  missions.  The  basis  for  NORTHCOM’s  exercises  is  DOD’s  Joint 
Training  System.  NORTHCOM’s  Training  and  Exercise  Directorate  is 
responsible  for  planning  and  executing  joint  training,  exercises,  and 
education  programs  to  ensure  NORTHCOM  is  prepared  to  accomplish  its 
assigned  missions. 


DOD’s  Joint  Training  Due  to  the  need  to  prepare  for  and  conduct  military  operations  to  defend 

System  the  United  States  and  fight  the  nation’s  wars,  DOD  has  developed  an 

established,  authoritative,  l  ime-test  ed  process  for  planning,  conducting, 
and  evaluating  exercises  in  order  to  test  and  improve  preparedness  to 
meet  its  wide  range  of  critical  missions.36  NORTHCOM  uses  DOD’s  Joint 
Training  System  as  the  basis  to  design,  develop,  and  conduct  exercises.37 
The  Joint  Training  System  provides  an  integrated,  requirements-based 
method  for  aligning  training  programs  with  assigned  missions  consistent 
with  command  priorities,  capabilities,  and  available  resources.  The  joint 
system  consists  of  four  phases  beginning  with  the  identification  of  critical 
capabilities  required  based  on  assigned  missions,  proceeding  through  the 
planning  and  scheduling  of  training  events,  the  execution  and  evaluation 
of  required  training,  and  assessing  training  proficiency  against  required 
capability  (see  fig.  3).  This  process  is  designed  to  ensure  that  an 
organization’s  training  program  is  linked  to  the  Joint  Mission  Essential 


35For  purposes  of  this  report,  large-scale  exercises  are  those  involving  multiple  agencies 
and  organizations,  venues,  and  events,  such  as  NORTHCOM’s  Vigilant  Shield  and  Ardent 
Sentry  exercises. 

36DOD  established  the  Joint  Training  System  in  fiscal  year  1994.  GAO,  Military  Training: 
Actions  Needed  to  Enhance  DOD’s  Program  to  Transform  Joint  Training,  GAO-05-548 
(Washington,  D.C.:  June  21,  2005)  3,  19. 

3'The  documents  that  provide  the  written  framework  for  the  Joint  Training  System  are 
Chainnan  of  the  Joint  Chiefs  of  Staff,  Instruction  3500.01E,  Joint  Training  Policy  and 
Guidance  for  the  Armed  Forces  of  the  United  States  (Washington,  D.C.:  May  31,  2008); 
Manual  3500.03B,  Joint  Training  Manual  for  the  Armed  Forces  of  the  United  States 
(Washington,  D.C.:  Aug.  31,  2007);  and  Manual  3500. 04E,  Universal  Joint  Task  Manual 
(Washington,  D.C.:  Aug.  25,  2008). 


Page  17 


GAO-09-849  Homeland  Defense 


Task  List,38  the  personnel  executing  the  tasks  are  properly  trained,  and 
shortfalls  in  training  are  identified  and  corrected  in  order  to  improve 
readiness.  The  Joint  Training  Information  Management  System  is  an 
automated  system  specifically  designed  to  assist  users  in  managing 
elements  of  each  of  the  four  phases  of  the  Joint  Training  System.39  During 
the  execution  phase,  commanders  and  directors  focus  on  executing  and 
evaluating  planned  training  events,  which  can  be  accomplished  through 
academic  training,  exercises,  or  a  combination  of  these  activities.40  During 
the  execution  stage  of  the  Joint  Training  System,  the  Joint  Event  Life 
Cycle  provides  a  five-stage  methodology  for  joint-event  development 
design,  planning,  preparation,  execution,  and  evaluation.  For  example, 
DOD  components  prepare  for  the  execution  of  an  exercise  by  conducting 
five  conferences,  such  as  the  Concept  Development  Conference  where 
exercise  and  training  objectives  are  discussed  and  scenarios  developed. 
Activities  for  the  Joint  Event  Life  Cycle  are  managed  through  the  Joint 
Training  Information  Management  System. 


38The  Joint  Mission  Essential  Task  List  outlines  those  tasks  that  are  essential  to  a 
combatant  command’s  ability  to  perform  assigned  missions.  The  list  is  a  key  element  in 
readiness  assessment  and  reporting. 

"Specifically,  the  Joint  Training  Information  System  provides  the  joint  community  with  an 
integrated  software  capability  to  identify,  collect,  analyze,  store,  and  disseminate  the  data 
required  to  execute  the  Joint  Training  System. 

".Joint  exercises  have  been  characterized  by  some  form  of  multiechelon,  computer-assisted 
exercises  which  embody  complex  simulation.  These  may  be  either  a  field  training  exercise, 
command  postexercise,  or  a  tabletop  exercise. 
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Figure  3:  DOD’s  Joint  Training  System 


Joint  Training  System 


Evaluating  lessons  learned  and  identifying  issues  for  corrective  actions  are 
fundamental  components  of  DOD’s  training  and  exercise  process.  The 
Chairman  of  the  Joint  Chiefs  of  Staff  provides  policy,  direction,  and 
guidance  for  DOD’s  Joint  Lessons  Learned  Program.41  The  objectives  of 
this  program  are  to  collect  and  analyze  observations  from  exercises  and 
real  world  events;  disseminate  validated  observations  and  findings  to 
appropriate  officials;  identify  and  implement  corrective  actions;  and  track 
corrective  actions  until  reobserved  in  a  subsequent  exercise  or  event  to 


41Chairman  of  the  Joint  Chiefs  of  Staff,  Joint  Lessons  Learned  Program,  3150.25D,  C-l. 
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ensure  that  the  issue  has  been  successfully  resolved.  Combatant 
commands,  including  NORTHCOM,  execute  lessons  discovery,  knowledge 
development,  and  implementation  activities  scaled  to  meet  the  command’s 
requirements  while  supporting  and  feeding  into  the  Chairman’s  Joint 
Lessons  Learned  Program  by  identifying  lessons  applicable  across 
combatant  commands  and  the  services. 


The  National  Exercise 
Program 


The  NEP  was  established  in  April  2007  under  the  leadership  of  the 
Secretary  of  Homeland  Security  to  prioritize  and  coordinate  federal,  state, 
and  local  exercise  activities  and  serves  as  the  principal  mechanism  for 
examining  the  preparation  of  the  federal  government  to  respond  to  an 
incident  and  adopting  policy  changes  to  improve  such  preparation.42  The 
day-to-day  staff-level  coordination  of  the  NEP  is  managed  by  the  NEP 
Executive  Steering  Committee — a  working  group  of  the  White  House’s 
Domestic  Readiness  Group  Exercise  and  Evaluation  Sub-Policy 
Coordination  Committee — and  is  chaired  and  facilitated  by  FEMA’s 
National  Exercise  Division.  The  steering  committee  is  also  responsible  for 
framing  issues  and  recommendations  for  the  full  coordination  committee 
on  exercise  themes,  goals,  objectives,  scheduling,  and  corrective  actions.43 
Figure  4  illustrates  the  major  events  and  milestones  of  the  NEP  and 
NORTHCOM’s  exercise  program  and  table  1  provides  information  on 
related  major  documents. 


42DHS,  National  Exercise  Program  Implementation  Plan. 

43Other  steering  committee  members  include  DOD,  the  Office  of  the  Director  of  National 
Intelligence,  the  Federal  Bureau  of  Investigation,  and  the  Departments  of  State,  Justice, 
Energy,  Transportation,  and  Health  and  Human  Services. 


Page  20 


GAO-09-849  Homeland  Defense 


Figure  4:  National  Exercise  Program  and  U.S.  Northern  Command  Exercise  Program  Time  line:  Major  Events  and  Milestones 
(2001-2009) 
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2004 
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•  United  Defense  03  Exercise 
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Hurricane 
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Learned 
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2008 


DOD  NEP 
Instruction 
issued 


2009 


•  National  Response  Plan  Published 


•  Vigilant  Shield  06  Exercise 

•  Ardent  Sentry  06  Exercise 

•  Vigilant  Shield  07  National  Exercise  Program  (NEP)  Charter  signed 

•  Revised  Homeland  Security  Exercise  and  FEMA  reorganizes 

Evaluation  Program  Guidance  published  . 

National  Exercise  Program  Implementation  Plan  approved 

•  Ardent  Sentry  07  .  Na(jona|  Preparedness  Guidelines  published 

National  Level  Exercise  1-08  (Top  Officials  4  +  Vigilant  Shield  08) 

•  National  Response  Framework  published 

National  Level  Exercise  2-08  (Ardent  Sentry  08) 

•  NORTHCOM  exercise 
guidance  issued 

•  Vigilant  Shield  09 


Source:  GAO  Analysis  of  DOD  and  DHS  documentation. 
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Table  1:  Major  Documents  Related  to  the  National  Exercise  Program  and  NORTHCOM 


Title 

Description 

Homeland  Security  Presidential  Directive-8 

Requires  the  development  of  the  national  preparedness  goal;  preparedness 
priorities;  an  assessment  system;  a  training  and  exercise  program;  a  system  to 
collect,  analyze,  and  disseminate  information  from  exercises,  training  events,  and 
actual  incidents;  equipment  standards;  and  a  federal  response  capability 
inventory. 

National  Response  Framework 

Establishes  a  comprehensive,  national,  all-hazards  approach  to  domestic  incident 
response.  Replaced  the  National  Response  Plan. 

National  Preparedness  Guidelines 

Establishes  a  vision  for  national  preparedness  and  provide  a  systematic  approach 
for  prioritizing  preparedness  efforts  across  the  United  States 

National  Planning  Scenarios 

Provides  planning  tools  that  represent  a  minimum  number  of  credible  scenarios 
depicting  the  range  of  potential  terrorist  attacks  and  natural  disasters  and  related 
impacts  facing  the  United  States  They  form  a  basis  for  coordinated  federal 
planning,  training,  and  exercises. 

U.S.  North  American  Aerospace  Defense 
Command  (NORAD)-NORTHCOM  Instruction 
10-156 

Establishes  the  NORAD  and  NORTHCOM  exercise  program,  provides  an 
overview  of  the  exercise  program,  describes  the  principal  NORAD  and 
NORTHCOM-sponsored  exercises,  and  outlines  the  process  for  scheduling, 
planning,  execution,  and  afteraction  review  of  exercises. 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 
3150.25D 

Codifies  the  Joint  Lessons  Learned  Information  System  as  the  DOD  system  of 
record  for  the  Joint  Lessons  Learned  Program. 

DOD  Instruction  3020.47 

Establishes  policy,  assigns  responsibilities,  and  provides  instructions  for  DOD 
participation  in  the  National  Exercise  Program. 

Source:  DOD  and  DHS. 


The  NEP  includes  a  series  of  national  exercises  projected  on  a  5-year 
exercise  schedule.  These  exercises  are  organized  into  four  tiers  with  each 
tier  reflecting  different  requirements  for  interagency  participation  (see  fig. 
5). 
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Figure  5:  Description  of  NEP  Tiers 


Tier  III 


Tier  IV 


White  House  directed,  U.S.  Government-wide 
strategy  and  policy  focus,  full  federal  agency 
participation.  5  Tier  I  excercises  conducted  annually 
(1  national-level  exercise  and  4  principal-level 
exercises). 

Federal  Department/Agency  directed, 

U.S.  government-wide  strategy  and  policy  focus.  Full 
Federal  Agency  participation  in  person  or  through 
simulation.  No  more  than  3  Tier  II  exercises  conducted 
annually. 

Other  federal  exercises,  regional  operational,  tactical  or 
organizational-specific  focus.  Full  federal  agency 
participation  is  optional.  The  number  of  exercises 
conducted  annually  depends  upon  individual  entities’ 
needs  and  requirements. 

Non-Federal  Exercises,  State,  Territorial,  Local  and  or 
tribal  governments  and/or  private  sector  focus.  Federal 
Agency  participation  is  optional.  The  number  of 
exercises  conducted  annually  depends  upon  individual 
entities'  needs  and  requirements. 


Source:  FEMA. 


FEMA  administers  the  NEP  and  maintains  the  Homeland  Security 
Exercise  and  Evaluation  Program — a  capabilities  and  performance-based 
exercise  program — to  provide  standardized  policy,  methodology,  and 
terminology  for  exercise  design,  development,  conduct,  evaluation,  and 
improvement  planning.  DHS  maintains  policy  and  guidance  for  this 
program.44 

Similar  to  DOD’s  Joint  Training  System,  the  Homeland  Security  Exercise 
Evaluation  Program  uses  an  exercise  life  cycle  with  five  phases: 
foundation,  design  and  development,  conduct,  evaluation,  and 
improvement  planning.  This  program  also  provides  document  templates 
for  exercise  planning  and  evaluation  and  a  collection  of  interactive,  on-line 
systems  for  exercise  scheduling,  design,  development,  conduct, 
evaluation,  and  improvement  planning,  referred  to  as  the  Homeland 


44DHS,  Homeland  Security  Exercise  and  Evaluation  Program,  Volumes  1-5  (Washington, 
D.C.:  February  2007). 
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Security  Exercise  Evaluation  Program  Tool  Kit  (see  fig.  6). 45  FEMA  also 
has  additional  resources  to  support  exercises.  For  example,  exercise 
stakeholders  can  access  FEMA’s  Lessons  Learned  Information  Sharing 
system,  an  interagency  Web  site  for  posting  lessons  learned  and  sharing 
best  practices,  to  learn  about  promising  practices  that  could  facilitate 
exercise  activities.46 


45https://hseep.dhs.gov/pages/1001_Toolk.aspx. 

46https://llis.gov. 
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Figure  6:  Homeland  Security  Exercise  and  Evaluation  Program  Tool  Kit 


Source:  GAO  analysis  of  FEMA  documentation. 
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NORTHCOM’s 
Exercise  Program  Is 
Consistent  with 
DOD’s  Joint  Training 
System,  but  Exercise 
Reporting  is 
Inconsistent 


NORTHCOM’s  Exercise 
Program  is  Consistent  with 
DOD’s  Joint  Training 
System 


NORTHCOM’s  Commander’s  Training  Guidance  requires  that  NORTHCOM 
establish  a  training  and  exercise  program  consistent  with  the  Joint 
Training  System  and  establishes  that  training  efforts  and  resources  will  be 
focused  on  two  large-scale  exercises  annually.47  The  Joint  Training  System 
requires,  among  other  things,  that  an  organization’s  training  objectives  be 
linked  to  its  Joint  Mission  Essential  Task  List  and  include  the  use  of  the 
Joint  Events  Life  Cycle  for  planning,  conducting,  and  assessing  exercises.48 
We  found  that  NORTHCOM  has  developed  a  comprehensive  exercise 
program  consistent  with  DOD’s  Joint  Training  System.  For  example, 
NORTHCOM  uses  the  Joint  Training  Information  Management  System  to 
link  training  objectives  with  its  Joint  Mission  Essential  Task  List. 
NORTHCOM  officials  enter  information  on  task  performance  of  exercise 
participants  into  the  Joint  Training  Information  Management  System  to 
evaluate  the  extent  to  which  the  command  is  trained  based  on 
performance  requirements  in  the  Joint  Mission  Essential  Task  List.49 
NORTHCOM  also  uses  the  Joint  Training  Information  Management  System 
to  manage  the  Joint  Events  Life  Cycle  for  its  large-scale  exercises, 
including  planning  exercise  milestones  and  developing  a  time  line  that 


47U.S.  Northern  Command,  Commander’s  Training  Guidance  for  FY  09-10,  A-ll. 

48The  Joint  Events  Life  Cycle  provides  a  sequence  of  the  inputs,  process  steps,  and  outputs 
necessary  to  execute  any  size  exercise.  This  process  consists  of  five  stages  including 
design,  planning,  preparation,  execution,  and  evaluation.  A  series  of  planning  conferences 
provide  discrete  breakpoints  between  each  stage.  Chairman  of  the  Joint  Chiefs  of  Staff, 
Joint  Training  Manual  for  the  Armed  forces  of  the  United  States,  E-6. 

49NORTHCOM  is  given  an  overall  rating  in  the  system  for  the  command’s  current  status  of 
“trained”,  “needs  practice,”  or  “untrained”  for  each  Joint  Mission  Essential  Task.  U.S. 
Northern  Command,  Memorandum  on  NORAD  and  USNORTHCOM  Joint  Training  Plan 
FY08-09( Colorado  Springs,  Colo.:  July  30,  2007,  A-32-A-35;  Chairman  of  the  Joint  Chiefs  of 
Staff,  Joint  Training  Manual  for  the  Armed  Forces  of  the  United  States,  F-5-F-6. 
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allows  exercise  planners  to  see  where  they  are  in  the  event  life-cycle 
process.  For  example,  NORTHCOM  holds  five  planning  conferences  for 
each  exercise,  including  a  concept  development  conference,  where 
exercise  and  training  objectives  are  discussed  and  scenarios  developed.50 
We  also  found  that  NORTHCOM  has  conducted  13  large-scale  exercises 
since  it  was  created  in  2002,  generally  including  2  exercises  each  year  (see 
table  2).  Vigilant  Shield  is  held  in  the  fall  and  focuses  primarily  on 
NORTHCOM’s  homeland  defense  mission,  and  Ardent  Sentry  is  generally 
conducted  in  the  spring  and  focuses  on  defense  support  of  civil 
authorities. 


Table  2:  Large-Scale  NORTHCOM  Exercises  Since  2002 


Exercise 

Dates 

Scenarios 

Ardent  Sentry  09 

June  18-24,  2009 

Flooding  along  Mississippi  River 

Domestic  terrorist  organization  releases  foot  and  mouth  disease  at  4-H  expo 
Anthrax  released  into  the  food  supply 

Train  derailments  in  Iowa  and  Kansas  resulting  in  chemical  releases 

Domestic  terrorist  organization  launches  attack  against  a  Wyoming  Air  Force 
Base 

U.S.  and  Canadian  maritime  incidents 

Vigilant  Shield  09 

Nov.  12-18,  2008 

Earthquake  along  San  Andreas  Fault 

Flomeland  defense  vignettes  associated  with  mission  areas 

Ardent  Sentry  08 

May  1-8,2008 

Category  4  hurricane  strike  and  terrorist  threat  affecting  National  Capital 
Region 

Accidental  VX  nerve  gas  release  in  Oregon 

Multiple  terrorist  attacks  in  Washington  to  exercise  both  response  to  chemical 
weapons  attack  and  specific  defense  support  of  civil  authorities  capabilities 

Vigilant  Shield  08 

Oct.  15-20,  2007 

Series  of  exploded  radiological  dispersal  devices 

Aircraft  accidents 

Reverse  no-fly  list 

Deployment  of  a  command  assessment  element 

Ardent  Sentry  07 

April  30  -  May  17,2007 

Critical  infrastructure  protection 

Maritime  operations 

Detonation  of  a  10-kiloton  improvised  nuclear  device  in  Indiana 

Category  III  hurricane  impacting  the  New  England  area 

o0Chairman  of  the  Joint  Chiefs  of  Staff,  Joint  Training  Manual  for  the  Aimed  Forces  of  the 
United  States,  E-6-E-12. 
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Exercise 

Dates 

Scenarios 

Vigilant  Shield  07 

Oct.  16-27,  2006  and 
Dec.  4-15,  2006 

•  Crash  and  explosion  of  plane  carrying  nuclear  weapons 

•  Maritime  attacks/events 

•  Rogue  aircraft  and  North  American  Air  Defense  Command  Air  and  Strategic 
Threat  events 

•  Strategic  Ballistic  Missile  Defense  Force  Protection 

•  Defense  against  cyber  attacks 

Ardent  Sentry  06 

May  8-18,  2006 

•  Hurricane  strike  on  New  Orleans,  Louisiana 

•  Explosion  of  a  railcar  carrying  phosgene 

•  U.S.  and  Canadian  Infrastructure  attack 

•  Chemical  attack 

•  Radiological  incident 

•  Avian  flu  pandemic  outbreak  scenario 

Vigilant  Shield  06 

Nov.  1-10,  2005 

•  Chemical  scenario 

•  Critical  infrastructure  protection  of  a  nuclear  power  plant 

•  Biological  scenario 

•  Ballistic  missile  defense  threat 

•  Aerospace  defense 

Ardent  Sentry  05 

Apr.  4-9,  2005 

•  Exercise  deployment  and  employment  of  Joint  Task  Force-Civil  Support  in 
support  of  FEMA  Region  1  mission  assignment  requests  in  a  chemical 
response 

•  Response  to  a  biological  event 

•  Protection  of  critical  infrastructure  (nuclear  power  plant) 

Determined  Promise  04 

Aug.  5-10,  2004 

•  Simultaneous  terrorist  air  events  (hijacked  aircraft,  aircraft  with  terrorists  on 
watch  list,  low-altitude  defense  and  swarm  attack  by  business  jet  aircraft) 

•  Explosion  of  container  with  radiological  dispersal  device  in  Long  Beach, 
California  harbor 

•  Train  derailment  and  chemical  spill 

•  Multiple  sarin,  mustard,  and  chlorine  gas  attacks  in  Virginia 

•  Maritime  intercept  operations 

•  Attack  by  unmanned  aerial  vehicles  with  high-explosive  warheads  fired  from 
oil  platform 

•  Cyber  security  event 

•  Ballistic  missile  defense 

Unified  Defense  04 

Feb.  19-25,  2004 

•  Air  threats  to  Alaska,  Canada,  and  the  United  States 

•  Category  IV+  hurricane  in  Corpus  Christi,  Texas 

•  Nuclear  detonation  in  Texas 

•  Quick  reaction  force  deployments  to  Texas  and  Alaska 

•  Transportation  security  and  U.S.  Coast  Guard  incidents  in  Alaska 

•  Maritime  intercept  operations  events 

•  Ballistic  missile  defense  threat 
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Exercise 

Dates 

Scenarios 

Determined  Promise  03 

Aug.  18-28,  2003 

Terrorist  attack  of  a  biological  weapon  of  mass  destruction  in  Clark  County, 
Nevada 

Wildfires  in  the  western  United  States 

2003  World  Gymnastics  Championship  in  California 

Ongoing  operational  plan  load-out  operations 

Sustained,  diverse  maritime  events 

Air  threat  to  Alaska 

High-interest  vessel  threat 

Strategic  infrastructure  threats  in  the  northwest  United  States 

Category  III  hurricane  in  the  southeast  United  States 

Train  derailment  in  Kentucky 

Shipping  and  security  issues  in  Alaska 

Unified  Defense  03 

Feb.  6-13,  2003“ 

National  special  security  event-  SuperBowl 

J  ointTask  Force-Civil  Support  held  in  strategic  reserve  for  State  of  the  Union 
Address  and  SuperBowl 

Nuclear  threat  along  southwestU.S.  border 

Border  surveillance 

Threat  to  port 

Consequence  management  in  northeast  United  States 

Natural  disaster  in  Alaska 

Source:  GAO  analysis  of  NORTHCOM  documents. 

“Unified  Defense  03  started  late  due  to  the  Columbia  Shuttle  disaster  and  ended  early  due  to  a  level 
Orange  alert  in  the  Homeland  Security  AlertSystem. 


NORTHCOM’s 
Documentation  of 
Exercises  Is  Generally 
Timely  but  Inconsistent 


NORTHCOM  guidance  outlines  the  postexercise  documentation  required 
to  be  completed  for  each  exercise,  including  quick  look,  after-action,  and 
exercise  summary  reports;  provides  a  time  line  for  the  completion  of  these 
documents;  and  includes  general  direction  that  these  documents  follow 
the  same  focus  areas  as  the  collection  management  plan — the  source 
document  from  which  exercise  analysts  identify,  examine,  and 
recommend  emerging  issues  and  trends.51  We  found  that  NORTHCOM  has 
generally  completed  exercise  summary  reports  for  its  exercises;  however, 
neither  NORTHCOM  nor  Joint  Forces  Command  officials  could  locate  an 
exercise  summary  report  for  Unified  Defense  03. 52  In  addition, 


olU.S.  Northern  Command,  Operations:  Exercise  Program,  Instruction  10-156  (Colorado 
Springs,  Colo.:  June  2008),  20. 

52While  NORTHCOM  was  unable  to  provide  exercise  summary  or  quick-look  reports  for 
United  Defense  03,  they  did  provide  pre-exercise  briefings.  NORTHCOM  officials  told  us 
that  the  exercise  started  late  due  to  the  Columbia  shuttle  disaster  and  ended  early  due  to  a 
level  Orange  alert  in  the  Homeland  Security  Alert  System. 
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postexercise  documentation  is  not  consistently  included  on  NORTHCOM’s 
portal  or  the  Joint  Training  Information  Management  System.  NORTHCOM 
guidance  issued  in  June  2008  provides  a  time  line  for  the  completion  of 
postexercise  documents  and  has  been  applicable  to  2  subsequent 
exercises-Ardent  Sentry  08  and  Vigilant  Shield  09.  According  to  the  2008 
guidance,  the  exercise  summary  report  is  to  be  submitted  to  the 
NORTHCOM  Commander  within  90  days  of  completing  an  exercise.63  The 
Ardent  Sentry  08  and  Vigilant  Shield  09  exercise  summary  reports  were 
issued  99  days  and  92  days,  respectively,  after  the  completion  of  each 
exercise.  Overall,  we  reviewed  exercise  summary  reports  for  11  of 
NORTHCOM’s  large-scale  exercises  that  have  taken  place  since  2003. 
Seven  of  the  11  exercise  summary  reports  were  issued  within  100  days. 
Four  of  the  reports  were  issued  later  than  100  days,  and  1  of 
NORTHCOM’s  earlier  reports  was  issued  in  less  than  30  days. 

NORTHCOM  guidance  states  that  exercise  summary  reports  should 
provide  the  official  description  of  the  exercise,  identify  significant  lessons 
learned,  and  be  targeted  toward  a  national  audience.54  Guidance  also 
requires  that  exercise  summary  reports  follow  the  same  focus  areas  as  the 
collection  management  plan — the  source  document  from  which  exercise 
analysts  identify,  examine,  and  recommend  emerging  issues  and  trends.65 
We  found  that  NORTHCOM’s  exercise  summary  reports  generally  included 
an  executive  summary,  training  objectives,  and  the  exercise’s  major 
scenarios  and  events,  but  did  not  consistently  include  lessons  learned, 
exercise  strengths  and  weaknesses,  or  clear  recommendations.  The 
exercise  summary  reports  that  included  a  section  on  lessons  learned 
lacked  details.  For  example,  6  of  the  11  exercise  summary  reports  we 
reviewed  included  an  identified  lessons  learned  section,  and  just  1  of  these 
6  reports — Unified  Defense  04 — provided  additional  information  on 
lessons  learned  beyond  identifying  the  title  of  each  observation  and  the 
status  of  the  observation  in  the  lessons  learned  management  system.  As 


53U.S.  Northern  Command,  Operations:  Exercise  Program.  The  instruction,  issued  in  June 
2008,  also  requires  exercise  quick  look  reports  be  issued  within  30  days  after  the 
completion  of  an  exercise.  The  instruction  is  only  applicable  to  Ardent  Sentry  08  and 
Vigilant  Shield  09,  as  the  post-exercise  documents  for  these  reports  were  issued  after  June 
2008.  Post-exercise  documents  for  all  other  exercises  reviewed  by  GAO  were  issued  prior 
to  June  2008. 

54U.S.  Northern  Command,  Operations:  Exercise  Program,  20. 

55U.S.  Northern  Command,  Operations:  Exercise  Program,  20. 
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discussed  later  in  this  report,  access  to  this  system  is  required  in  order  to 
obtain  any  additional  information  on  the  lesson  learned.56 

We  also  found  that  NORTHCOM  exercise  summary  reports  have  not 
followed  the  same  focus  areas  as  collection  management  plans. 57  For 
example,  none  of  the  seven  exercise  summary  reports  for  NORTHCOM 
exercises  conducted  since  Hurricane  Katrina  in  2005  reported  on  the 
information  identified  in  the  collection  management  plans’  focus  areas. 
Inconsistencies  in  exercise  documentation  may  be  occurring  because  DOD 
and  NORTHCOM  guidance  do  not  require  a  standard  format  or  specific 
content  for  postexercise  documentation.  Although  NORTHCOM  uses  other 
methods  to  document  exercises,  such  as  the  Joint  Training  Information 
Management  System,  this  system  does  not  include  a  complete  record  of 
each  exercise.  For  example,  the  Joint  Training  Information  Management 
System  does  not  include  the  lessons  learned  from  an  exercise.  In  addition, 
access  to  this  system  is  generally  limited  to  DOD  officials.  Recognizing  the 
need  for  a  complete  and  consistent  record  of  each  exercise,  DHS’s 
Homeland  Security  Exercise  Evaluation  Program  provides  a  template  for 
exercise  documentation,  including  format  and  content.68  NORTHCOM 
used  this  template  for  National  Level  Exercise  2-08,  but  does  not  use  the 
template  for  its  own  exercises.  59  Despite  differences  in  the  requirements 
and  complexities  of  NORTHCOM’s  and  DHS’  exercise  programs,  the  lack 
of  a  complete  and  consistent  record  of  each  exercise  lessens  the  extent  to 
which  NORTHCOM  can  ensure  it  has  trained  to  key  focus  areas.  Further,  it 
deprives  the  command  of  a  key  source  of  historical  information  upon 
which  to  base  current  and  future  assessments  of  exercises  and  a 


56 Access  to  JLLIS  is  restricted  in  order  to  safeguard  sensitive  information  because  of 
concerns  that  adversaries  may  be  able  to  exploit  any  weaknesses  contained  in  those 
records.  For  users  to  access  the  unclassified  portion  of  the  JLLIS  they  must  have  a  DOD 
Common  Access  Card,  a  card  reader,  and  appropriate  computer  software  prior  to 
requesting  access. 

57Collection  Management  Plans  are  generally  linked  to  the  Joint  Mission  Essential  Tasks 
and  exercise  objectives.  Focus  areas  include  a  wide  range  of  operations,  such  as 
coordinating  and  integrating  regional  interagency  activities  and  coordinating  consequence 
management  in  theater.  U.S.  Northern  Command,  Operations:  Exercise  Program,  20. 

58DHS,  Homeland  Security  Exercise  Evaluation  Program.,  Volume  IIP.  Exercise 
Evaluation  and  Implementation  Plan,  (Washington,  DC:  February  2007),  A-l  -  B-2. 

59U.S.  Northern  Command,  North  American  Aerospace  Defense  Command  and  United 
States  Northern  Command  National  Level  Exercise  2-08: 1  -8  May  2008  Exercise 
Summary  Report,  (Colorado  Springs,  Colo.:  Aug.  15,  2008)1-25. 
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consistent  venue  for  sharing  lessons  learned  with  interagency  partners  and 
states. 


Interagency  Partners 
and  States  Have 
Participated  In 
NORTHCOM’s 
Exercises,  but 
Challenges  Remain 
with  Including  States 
in  Planning, 

Conducting,  and 
Assessing  Exercises 

Federal  Agencies  and  NORTHCOM  recognizes  the  importance  of  exercising  with  key  partners  in 

States  Have  Participated  in  all  its  missions  and  that,  in  order  to  achieve  its  goal  of  being  trained  and 
NORTHCOM  Exercises  ready  to  execute  joint  operations  and  ensure  a  seamless  operating 

environment,  NORTHCOM  should  maximize  exercise  participation  with 
federal,  state,  and  local  agencies  and  National  Guard  units.60  NORTHCOM 
has  included  interagency  partners,  such  as  DHS,  FEMA,  and  the  U.S.  Coast 
Guard,  and  several  states  in  its  large-scale  exercises  (see  table  3).  We 
found  that  17  civilian  federal  agencies  and  organizations  have  participated 
to  varying  degrees  in  one  or  more  of  the  seven  large-scale  NORTHCOM 
exercises  that  have  occurred  since  Hurricane  Katrina  made  landfall  in 
August  2005.  Seventeen  states  have  participated  in  NORTHCOM  exercises 
since  that  time,  and  8  of  these  states — Arizona,  California,  Connecticut, 
Indiana,  Michigan,  Oregon,  Rhode  Island,  and  Washington — played  a 
major  role  by  having  a  portion  of  the  exercise  conducted  in  the  state  and 


60U.S.  Northern  Command,  Comm, cinder’s  Training  Guidance  for  FY09-1 0;  Chairman  of 
the  Joint  Chiefs  of  Staff,  Joint  Training  Manual  for  the  Armed  Forces  of  the  United 
States,  CJCSM  3500.03B  (Aug.  31,  2007). 
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having  various  state  agencies  and  officials  participate.61  For  example, 
Indiana  and  Rhode  Island  played  major  roles  in  Ardent  Sentry  07  for  the 
detonation  of  a  10-kiloton  improvised  nuclear  device  and  category  III 
hurricane  impacting  the  New  England  region,  respectively.  Both  states 
established  emergency  operating  centers  and  exercised  large  numbers  of 
state  emergency  management  personnel. 


Table  3:  Interagency  Partner  and  State  Participation  in  Large-scale  NORTHCOM  Exercises  Since  2006 


Other  federal  agencies 

Exercise 

and  organizations 

States 

Vigilant  Shield  06 

Department  of  Homeland  Security 

•  Connecticut 

Other  federal  organizations  in  the  areas  of  homeland  defense 
and  defense  support  of  civil  authorities6 

•  New  Jersey 

Ardent  Sentry  06 

Coast  Guard  Sector  Detroit  District  9 

•  Arizona 

Department  of  Energy 

-  Division  of  Emergency 

Department  of  Health  and  Human  Services 

Management 

Department  of  Homeland  Security 

-  Joint  Terrorist  Task  Force 

Department  of  Justice 

-  Tactical  Intelligence  Center 

Department  of  State 

•  Maine 

Department  of  Transportation 

•  Michigan 

Environmental  Protection  Agency 

-  Detroit  Metropolitan  Airport 

Federal  Aviation  Administration 

-  State  Emergency  Operation 

Federal  Bureau  of  Investigation 

Center 

FEMA 

-  State  Police 

Transportation  Security  Agency 

-  Service  Emergency 

U.S.  Coast  Guard/Atlantic  Area 

Preparedness  Liaison  Officer 

U.S.  Department  of  Agriculture 

-  St.  Clair  County  Regional 
Response  Team 

-  Wayne  County 

61We  visited  all  of  these  states  except  for  Connecticut  and  Indiana.  We  visited  Indiana 
during  a  review  of  NORTHCOM’s  planning  and  interagency  coordination  in  the  spring  of 
2007.  We  also  met  with  Nevada  officials  who  participated  in  Determined  Promise  03.  The 
remaining  9  of  the  17  states  played  lesser  roles  in  the  exercises. 
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Exercise 

Other  federal  agencies 
and  organizations 

States 

Vigilant  Shield  07 

• 

Department  of  Energy 

• 

Arizona 

• 

DHS  National  Interagency  Simulation  Cell 

-  Pima  County  Emergency 

• 

Department  of  Justice 

Management 

• 

Department  of  Transportation 

-  Tucson  Police  and  Fire 

• 

FEMA 

• 

Washington,  DC 

• 

DHS  Principal  Federal  Official 

• 

Alaska 

• 

U.S.  Coast  Guard 

• 

Department  of  Agriculture 

• 

Environmental  Protection  Agency 

• 

Federal  Bureau  of  Investigation 

• 

Department  of  Health  and  Human  Services 

• 

DHS  Immigration  and  Customs  Enforcement 

• 

National  Geospatial  Intelligence  Agency 

• 

DHS  Transportation  Security  Administration 

• 

National  Security  Agency 

• 

Veterans  Administration 

Ardent  Sentry  07 

• 

DHS-National  Operations  Center 

• 

Indiana 

• 

FEMA 

-  Department  of  Homeland  Security 

• 

FEMA  Region  1 

-  Indianapolis,  Indiana 

• 

FEMA  Region  II 

-  Marion  County,  Indiana 

• 

FEMA  Region  X 

• 

Rhode  Island 

• 

Federal  Bureau  of  Investigation 

• 

Connecticut 

• 

U.S.  Coast  Guard  Pacific  Area 

• 

Massachusetts 

• 

Vermont 

• 

New  Hampshire 

• 

Maine 

• 

New  York 

• 

Ohio 

• 

Illinois 

• 

Alaska 

National  Level 

• 

FEMA 

• 

Oregon 

Exercise  1-08/Vigilant 

• 

DHS-National  Operations  Center 

-  City  of  Portland 

Shield  08 

• 

FEMA  Regions  IX  and  X 

Arizona 

• 

U.S.  Coast  Guard  Pacific  Area 

-  City  of  Phoenix 

• 

Guam3 

National  Level 

• 

Department  of  Homeland  Security 

• 

Washington,  DC 

Exercise  2-08/Ardent 

• 

FEMA  National  Exercise  Division 

• 

Oregon 

Sentry  08 

• 

Other  federal  organizations  in  the  areas  of  homeland  defense 

• 

Washington 

and  defense  support  of  civil  authorities'1 
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Other  federal  agencies 

Exercise 

and  organizations 

States 

Vigilant  Shield  09 

•  FEMA  Region  IX 

•  Other  federal  organizations  in  the  areas  of  homeland  defense 
and  defense  support  of  civil  authorities6 

•  California 

Ardent  Sentry  09 

•  Dept  of  Energy 

•  Dept  of  Health  and  Human  Services 

•  Dept  of  Homeland  Security 

•  Environmental  Protection  Agency 

•  Federal  Aviation  Administration 

•  Federal  Bureau  of  Investigation 

•  FEMA  Region  VII 

•  FEMA  Region  VIII 

•  U.S.  Coast  Guard  Operational  Command 

•  FEMA  National  Response  Coordination  Center 

•  Iowa 

-  Dept  of  Agriculture 

-  Dept  of  Health 

-  Dept  of  Public  Safety 

-  Homeland  Security  &  Emergency 
Management  Division 

-  Iowa  State  Emergency 

Operations  Center 

-  City  of  Des  Moines  Fire  Dept. 

-  City  of  Des  Moines  Police  Dept. 
Kansas 


-  Dept  of  Emergency  Management 

-  Dept  of  Health/Environment 

-  Dept  of  Agriculture 

-  Dept  of  Animal  Health 

-  Dept  of  Transportation 

-  Highway  Patrol 

•  Nebraska 

•  Missouri 

•  Wyoming 


Source:  GAO  analysis  of  NORTHCOM  exercise  documents. 

"The  US  territory  of  Guam  is  in  U.S.  Pacific  Command’s  area  of  responsibility. 

"Exercise  summary  reports  for  Ardent  Sentry  08  and  Vigilant  Shield  06  and  09  indicate  that  other 
federal  organizations  in  the  areas  of  homeland  defense  and  defense  support  of  civil  authorities 
participated  in  the  exercises,  but  the  reports  do  not  provide  specific  names  of  organizations  and  we 
were  unable  to  identify  these  organizations  in  the  Joint  Training  Information  Management  System. 


State  emergency  management  and  National  Guard  officials  told  us  that 
they  participated  in  NORTHCOM  exercises  because  they  wanted  to  better 
understand  the  (1)  capabilities  that  NORTHCOM  could  bring  to  the 
response  to  an  incident  and  (2)  command  and  control  issues  of  the  troops 
in  a  state  when  NORTHCOM  is  involved.  We  previously  reported  that 
states’  participation  in  NORTHCOM  exercises  helps  to  build  relationships 
and  improve  coordination.62  Officials  from  all  of  the  states  we  met  with 


62GAO-08-252. 
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told  us  that  they  derived  benefits  from  their  participation  in  these 
exercises.  For  example,  state  emergency  management  officials  from  three 
states  told  us  that  first-hand  interaction  with  federal  military  forces  and 
the  opportunity  to  observe  the  federal  response  to  an  incident  was 
beneficial.  In  addition,  two  state  emergency  management  and  National 
Guard  officials  told  us  that  NORTHCOM  officials  were  professional,  well- 
trained,  and  helpful.  Further,  officials  from  five  states  told  us  that 
NORTHCOM  provided  beneficial  resources,  such  as  funds  for  travel  to 
attend  exercise  planning  conferences  and  contractor  staff  to  help  state 
officials  prepare  exercise  scripts  and  injects.63  Finally,  officials  from  two 
states  told  us  that  the  benefits  of  working  with  NORTHCOM  included 
gaining  an  understanding  of  the  resources  and  capabilities  that 
NORTHCOM  can  provide,  as  well  as  understanding  how  NORTHCOM 
coordinates  its  response  through  FEMA. 

NORTHCOM  is  also  attempting  to  include  states  in  exercises  through  the 
Vigilant  Guard  Program.  The  goal  of  the  Vigilant  Guard  Program  is  to 
enhance  National  Guard  and  State  emergency  management  agency 
preparedness  to  perform  their  homeland  defense  and  Defense  Support  to 
Civil  Authorities  roles  and  responsibilities.  It  focuses  on  State  Guard  Joint 
Force  Headquarters  coordination  with  the  state  emergency  management 
agency  and  Joint  Task  Force-State  operations  and  involves  multiple  states 
and  agencies.  The  program  began  in  September  2004  and  included  one 
exercise  in  fiscal  year  2005.  Now  the  plan  is  to  conduct  four  exercises 
annually.  NORTHCOM  was  given  management  responsibility  for  the 
Vigilant  Guard  exercises  in  2007,  although  the  National  Guard  Bureau 
retains  responsibility  for  budgeting  for  these  events.  Two  of  the  four 
annual  Vigilant  Guard  exercises  are  to  be  linked  to  major  combatant 
command  exercises,  usually  NORTHCOM’s  Ardent  Sentry  and  Vigilant 
Shield.  States  hosting  a  Vigilant  Guard  exercise  determine  the  objectives 
for  these  events,  and  NORTHCOM  provides  support.  Separate  planning 
begins  for  these  Vigilant  Guard  exercises  prior  to  the  related  planning 
meetings  for  any  linked  NORTHCOM  exercise.  NORTHCOM’s  Ardent 
Sentry  09  is  linked  with  a  Vigilant  Guard  exercise  in  Iowa  with  scenarios 
including  a  train  derailment  and  a  chemical  spill,  an  epidemic  outbreak, 
and  terrorism  incident. 


63  Injects  are  pieces  of  information  presented  to  the  training  audience  during  an  exercise  in 
a  manner  similar  to  real-world  operations  to  guide  the  training  audience  toward  the 
accomplishment  of  the  training  objectives. 
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NORTHCOM  Faces 
Challenges  Involving 
Interagency  Partners  and 
States  in  Planning, 
Conducting,  and  Assessing 
Exercises 


A  key  element  to  developing  effective  working  relationships  with  all  states 
is  a  well-thought  out  and  consistent  process  for  including  the  states  in 
planning,  conducting,  and  assessing  exercises.  Without  such  a  process, 
states  may  be  unwilling  to  participate  in  future  NORTHCOM  exercises, 
impacting  the  seamless  exercise  of  all  levels  of  government  and  potentially 
affecting  NORTHCOM’s  ability  to  provide  support  to  civil  authorities.  We 
found  that  challenges  remain  which  have  resulted  in  inconsistencies  in  the 
way  that  NORTHCOM  involves  the  states  in  its  exercises.  One  of  DOD’s 
challenges  is  adapting  its  exercise  system  and  practices  to  accommodate 
the  coordination  and  involvement  of  other  federal,  state,  local,  and  tribal 
agencies  that  do  not  have  the  same  kinds  of  practices  or  level  of  planning 
effort.  Differences  in  exercise  culture  stem  from  differences  in  missions, 
experience,  authority,  scope,  and  resources  available  to  DOD,  interagency 
partners,  and  states.  DOD  has  an  established,  authoritative,  time-tested 
process  for  planning,  conducting,  and  evaluating  exercises  in  order  to  test 
and  improve  preparedness  to  meet  its  wide  range  of  critical  missions. 
Within  DOD,  training  and  exercises  are  considered  a  vital  component  of  its 
overall  mission  of  defending  the  national  interests  and  significant 
resources  are  devoted  to  these  activities.  In  contrast,  DHS,  as  the  lead  for 
interagency  homeland  security  efforts,  is  a  new  agency  and  has  faced 
challenges  since  it  was  created  due  to  frequent  reorganization  and  not 
being  fully  staffed.64  DHS  and  other  civilian  agencies  and  state  and  local 
governments  have  day-to-day  missions  and  responsibilities  that  may  take 
priority  over  exercises  and  often  do  not  have  the  resources  or  experience 
to  participate  in  or  conduct  exercises.  For  example,  DOD  exercises  often 
are  conducted  24  hours  a  day,  7  days  a  week  and  may  last  a  week  or  more 
to  enhance  the  realism  of  the  exercise,  while  civilian  agencies  generally 
participate  8  hours  per  day,  usually — according  to  NORTHCOM  officials — 
during  normal  business  hours,  and  do  not  exercise  longer  than  a  few  days. 
Therefore,  DOD  exercises  are  generally  longer  in  duration,  more  resource 
intensive,  and  involve  more  participants  than  other  federal  and  state 
exercises.  Furthermore,  DOD  views  itself  as  the  last  line  of  defense  and 
often  exercises  until  resources  are  exhausted  to  fully  assess  capabilities 
and  identify  areas  needing  improvement.  Civilian  agencies  and  states  may 
prefer  not  to  exhaust  resources  during  an  exercise  in  order  to  avoid 
appearing  unprepared  for  an  incident  and  the  associated  political 
controversy. 


64Defense  Science  Board,  Unconventional  Operational  Concepts  and  the  Homeland,  viii, 
xiii,  38. 
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Another  challenge  that  NORTHCOM  faces  is  exercising  with  the  various 
states  and  territories  within  its  area  of  responsibility  considering  the  legal 
and  historical  limits  of  the  constitutional  federal-state  structure.  The  states 
have  a  wide  range  and  type  of  civilian  state  agencies  responsible  for 
emergency  management,  some  of  which  are  headed  by  the  Adjutant 
General  of  the  state,  who  also  heads  the  military  department  or  National 
Guard,  and  others  are  completely  separate  entities  (see  table  4). 


Table  4:  Emergency  Management  Structures  of  the  States  We  Visited 


State 

Exercise(s) 

Emergency  management  structure 

Arizona 

Ardent  Sentry  06 

Vigilant  Shield  07 

The  Adjutant  General  is  the  director  of  the  Department  of  Emergency  and  Military 
Affairs,  which  is  a  state  agency  combining  Arizona’s  Army  and  Air  National  Guard 
with  Joint  Military  Programs  and  Emergency  Management. 

California 

Vigilant  Shield  09 
Determined  Promise  4 

The  California  Emergency  Management  Agency  is  a  Governor's  cabinet-level 
agency  that  is  led  by  a  Secretary  who  reports  to  the  Governor. 

The  Adjutant  General  is  the  head  of  the  California  State  Military  Department 
under  which  the  National  Guard  operates. 

Michigan 

Ardent  Sentry  06 

The  Civilian  Emergency  Management  and  Homeland  Security  Division  is  located 
in  the  State  Police  Department. 

The  Adjutant  General  is  the  director  of  the  Department  of  Military  and  Veteran 
Affairs  under  which  the  National  Guard  operates. 

Nevada 

Determined  Promise  03 

The  Division  of  Emergency  Management  is  part  of  the  Department  of  Public 

Safety  (which  also  includes  the  Highway  Patrol  and  State  Fire  Marshall). 

The  Adjutant  General  is  the  head  of  the  Nevada  Office  of  the  Military  which  is  also 
known  as  the  National  Guard. 

Oregon 

Vigilant  Shield  08 

National  Level 

Exercise  1-08 

The  Oregon  Office  of  Emergency  Management  operates  administratively  under 
the  Adjutant  General  and  the  Oregon  Military  Department.  The  Adjutant  General 
also  directs,  manages,  and  supervises  the  Oregon  National  Guard.  The  Office  of 
Emergency  Management's  operations,  however,  are  kept  separate  from  those  of 
the  Oregon  National  Guard. 

Rhode  Island 

Ardent  Sentry  07 

The  Adjutant  General  heads  the  Emergency  Management  Agency  and  the 

National  Guard.  These  offices  are  co-located  with  the  National  Guard  playing  a 
supporting  role  to  the  Emergency  Management  Agency. 

Washington 

Ardent  Sentry  08 

National  Level 

Exercise  2-08 

The  Adjutant  General  heads  the  Military  Department  which  houses  the 

Washington  Emergency  Management  Division,  National  Guard,  Air  National 

Guard,  and  State  Guard. 

Source:  GAO  analysis  of  NORTHCOM  documents  and  information  provided  by  states. 


Working  with  states  has  been  the  responsibility  of  the  National  Guard  and 
is  relatively  new  for  a  federal  military  command  like  NORTHCOM. 
NORTHCOM  officials  face  challenges  in  dealing  with  the  various  civilian 
agencies,  differing  emergency  management  structures,  capabilities,  and 
needs  of  the  states.  For  example,  for  Ardent  Sentry  08  (linked  with 
National  Level  Exercise  2-08),  NORTHCOM  planned  a  scenario  involving  a 
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chemical  bomb  attack  in  Seattle,  Washington  without  consulting  the  state 
health  department  or  civil  support  team — the  agencies  responsible  for 
responding  to  a  chemical  or  biological  attack.  State  officials  told  us  that 
NORTHCOM  invited  the  health  department  to  participate  once  state 
officials  informed  them  that  they  should  be  involved,  but  that  the  scenario 
was  already  locked  in  without  the  input  of  this  key  participant.  DOD 
officials  told  us  that  they  rely  on  FEMA  regional  offices  to  provide 
information  on  state  agencies.  However,  we  believe  that  NORTHCOM 
officials  should  have  determined  if  all  relevant  agencies  were  included  in 
the  exercise  when  directly  interacting  with  state  officials  during  the 
scenario  development  and  other  planning  conferences,  before  the 
scenarios  were  locked  in.  Washington  emergency  management  officials 
told  us  that  this  affected  the  realism  of  the  exercise. 

NORTHCOM  also  faces  challenges  in  balancing  its  training  objectives  with 
those  of  state  agencies  and  organizations.  State  and  local  governments 
seek  to  exercise  their  first  responder  capabilities  before  having  their 
resources  overwhelmed  and  needing  to  seek  federal  assistance.  On  the 
other  hand,  NORTHCOM  seeks  to  exercise  its  capability  to  provide 
support  to  civil  authorities  when  local,  state,  and  other  federal  resources 
are  exhausted.  This  necessarily  requires  scenarios  that  exceed  the  states’ 
capabilities  and  that  stress  DOD  capabilities.  Officials  from  four  of  the 
seven  states  we  interviewed  told  us  that  NORTHCOM’s  exercise  scenarios 
appeared  unrealistic,  overwhelmed  their  states  too  soon  during  the 
exercise,  or  did  not  allow  states  to  fully  exercise  their  own  training 
objectives.65  For  example,  the  scenario  for  Ardent  Sentry  06  included 
multiple  improvised  explosive  devices  detonating  over  a  4-day  period  in 
various  sites,  such  as  the  City  of  Detroit,  St.  Clair  and  Wayne  Counties, 
Michigan,  and  Windsor,  Ontario,  Canada  with  over  14,000  fatalities  and  a 
simultaneous  pandemic  flu  outbreak  in  Michigan.66  State  emergency 
management  officials  told  us  that  such  a  large  number  of  casualties  would 
overwhelm  state  resources  almost  immediately,  and  therefore  precluded 


65Creating  an  exercise  scenario  event  that  is  believable  is  critical  to  engaging  participants  in 
a  level  of  play  and  experiential  learning  that  will  be  long  lasting.  Defense  Science  Board, 
Unconven  tional  Operational  Concepts  and  the  Homeland,  55. 

66U.S.  Northern  Command,  Northern  American  Aerospace  Defense  Command  and  United 
States  Northern  Command  Ardent  Sentry  06  Exercise  Summary  Report  (Colorado 
Springs,  Colo.:  Sept.  27,  2006),  4-6. 
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fully  exercising  training  objectives  for  state  and  local  responders.67 
Officials  from  these  states  told  us  that  because  they  did  not  have  the 
opportunity  to  exercise  their  own  training  objectives,  they  believed 
NORTHCOM  was  using  them  as  a  training  tool.  68  A  NORTHCOM  official 
told  us  that  NORTHCOM  needs  the  states  to  participate  in  exercises  and, 
therefore,  will  be  flexible  to  accommodate  other  organizations’  training 
objectives;  however,  NORTHCOM  ultimately  has  its  own  objectives  to 
exercise.  Officials  from  five  of  seven  states  noted  that,  for  example,  they 
face  budget  and  staffing  limitations,  and  playing  a  major  role  in  a 
NORTHCOM  exercise  often  requires  establishing  a  state  emergency 
operations  center  with  numerous  staff  and  agencies  involved.  Given  the 
expansive  scenarios  NORTHCOM  uses  to  guide  its  exercises  and  the 
perception  of  half  of  the  states  we  visited  that  this  limits  the  benefits  to 
them,  we  believe  that  the  states  may  be  less  likely  to  expend  scarce 
resources  to  participate  in  future  NORTHCOM  exercises. 

Inconsistencies  with  how  NORTHCOM  involves  states  in  planning, 
conducting,  and  assessing  exercises  is  occurring  in  part  because 
NORTHCOM  officials  lack  experience  dealing  with  the  various  state 
agencies  and  emergency  management  structures.  Inconsistencies  are  also 
occurring  because  NORTHCOM  has  not  established  an  informed, 
consistent  process  for  including  states  in  its  exercises.  One  aspect  of  this 
process  is  the  way  that  NORTHCOM  requests  state  participation  in  its 
exercises.  Currently,  NORTHCOM  has  various  processes  for  requesting 
that  other  federal  departments  and  agencies  participate  in  its  exercises, 
such  as  making  the  request  through  the  Joint  Staff.  FEMA  officials  told  us 
that  requests  for  state  participation  in  NORTHCOM  exercises  should  be 
made  through  FEMA’s  regional  offices.  However,  because  NORTHCOM 
does  not  have  an  established  process  for  requesting  state  participation, 
officials  from  the  states  we  visited  told  us  that  NORTHCOM  officials  made 
requests  informally  and  in  a  variety  of  ways,  including  through  the 
National  Guard  Bureau,  the  state’s  National  Guard,  or  FEMA’s  regional 


67State  emergency  management  officials  told  us  that  they  attended  planning  meetings  with 
NORTHCOM  and  raised  their  concern  about  the  scenarios,  but  NORTHCOM  officials  made 
the  decision  to  use  the  scenario  anyway  in  order  to  meet  their  own  training  objectives.  As  a 
result,  these  officials  told  us  that  several  other  states  dropped  out  of  the  exercise  for  this 
scenario  during  the  planning  stages. 

68The  Defense  Science  Board  recently  reported  that  national  exercises  typically  focus  on  a 
top-down  approach  where  the  supporting  organizations  are  training  aids  to  the  senior-level 
players,  instead  of  a  bottom-up  approach  focusing  on  an  integrated  and  layered  response. 
Defense  Science  Board,  Unconventional  Operational  Concepts  and  the  Homeland,  xv. 


Page  40 


GAO-09-849  Homeland  Defense 


offices.  69  In  some  cases,  such  as  when  the  state  emergency  management 
agency  and  state  National  Guard  have  a  close  working  relationship,  this 
method  has  been  effective  for  NORTHCOM.  However,  in  other  cases,  this 
method  has  led  to  more  limited  exercises.  For  example,  emergency 
management  officials  from  one  state  told  us  that  NORTHCOM  does  not 
have  full  state  representation  if  it  only  exercises  with  the  state  National 
Guard.  In  that  case,  NORTHCOM  therefore  misses  out  on  interaction  with 
other  key  state  emergency  management  officials  and  responders  and 
affects  the  realism  of  the  exercise. 

Another  aspect  of  the  lack  of  a  consistent  process  for  requesting  state 
participation  is  potentially  missing  the  opportunity  to  leverage  the  existing 
expertise  of  the  National  Guard  Bureau  and  defense  coordinating  officers 
located  in  each  of  the  10  FEMA  regional  offices.  As  we  previously 
reported,  the  National  Guard  Bureau  and  defense  coordinating  officers 
have  knowledge  and  experience  in  dealing  with  states  in  their  region  and 
may  be  a  valuable  resource  for  NORTHCOM  officials  during  the  planning 
and  conduct  of  exercises.70  The  three  defense  coordinating  officers  with 
whom  we  met  told  us  that  they  participate  in  NORTHCOM  exercises,  but 
currently  their  role  does  not  involve  requesting  state  participation  on 
behalf  of  NORTHCOM  or  providing  state-specific  information  to 
NORTHCOM  exercise  officials.  Without  an  informed  and  consistent 
process  for  including  the  states  in  planning,  conducting,  and  assessing  its 
exercises,  NORTHCOM  increases  the  risk  that  its  exercises  will  not 
provide  benefits  for  all  participants,  impacting  the  seamless  exercise  of  all 
levels  of  government  and  potentially  affecting  NORTHCOM’s  ability  to 
provide  support  to  civil  authorities. 


69In  some  cases,  NORTHCOM’s  coordination  with  the  National  Guard  has  been 
inconsistent.  For  example,  National  Guard  officials  from  one  state  told  us  that  it  is  imclear 
whether  communication  with  NORTHCOM  should  occur  directly  or  through  the  National 
Guard  Bureau. 

70Defense  Coordinating  Officers  are  senior-level  military  officers  who  provide  liaison 
support  and  serve  as  a  point  of  contact  for  federal,  state,  and  local  agencies  requiring  DOD 
support  during  an  incident.  GAO-08-252. 
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NORTHCOM  has  a 
Systematic  Lessons 
Learned  and 
Corrective  Action 
Program,  but  Gaps 
Remain 

DOD  and  NORTHCOM  guidance  requires  that  NORTHCOM  identify 
observations  during  the  course  of  normal  operations,  exercises,  and  real- 
world  events;  capture  the  detail  required  to  fully  understand  the  problem; 
and  share  valid  lessons  learned  and  issues  as  widely  as  possible.71 
NORTHCOM  has  been  identifying  observations,  lessons  learned,  and 
needed  corrective  actions  from  its  exercises  and  operations  since  the 
command  was  created  in  2002.  NORTHCOM  collects  and  tracks 
observations  through  the  Joint  Lessons  Learned  Information  System 
(JLLIS) — the  automated  official  DOD  system  for  managing  and  tracking 
exercise  observations  and  recording  lessons  learned.72  As  of  April  2009, 
DOD  exercise  participants  input  94  observations  into  JLLIS  during 
NORTHCOM’s  most  recent  large-scale  exercise,  Vigilant  Shield  09.  Table  5 
shows  the  observations  entered  into  JLLIS  or  its  predecessor  for 
NORTHCOM’s  major  exercises  since  2006.73 


NORTHCOM  Has  a 
Systematic  Lessons 
Learned  and  Corrective 
Action  Process 


''Chairman  of  the  Joint  Chiefs  of  Staff,  Joint  Lessons  Learned  Program,,  Instruction 
3150.25D,  A-l  and  A-6;  U.S.  Northern  Command,  Commander’s  Training  Guidance  FY  09- 
10,  A-4,  A-8,  A-20,  A-2-8:  U.S.  Northern  Command,  Lessons  Learned  Program  and 
Corrective  Action  Board  Process,  2. 

72The  Joint  Staff  develops  joint  lessons  learned  policy  and  serves  as  the  program  manager 
for  JLLIS. 

73 JLLIS  became  the  system  of  record  in  October  2008.  Prior  to  that  NORTHCOM’s  lessons 
learned  system  was  the  Advanced  Lesson  Management  System. 
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Table  5:  Number  of  Observations  Input  into  Lessons  Learned  Management  System  for  Large-Scale  Exercises  Since  2006 


NORTHCOM  exercise 

Unclassified 

observations3 

Classified 

observations 

Total 
observations 
per  exercise 

Vigilant  Shield  09 

75 

19 

94 

Ardent  Sentry  08 
(National  Level  Exercise  2-08) 

46 

0 

46 

Vigilant  Shield  08 
(National  Level  Exercise  1-08) 

34 

0 

34 

Ardent  Sentry  07 

113 

4 

117 

Vigilant  Shield  07 

95 

12 

107 

Ardent  Sentry  06 

54 

2 

56 

Vigilant  Shield  06 

12 

1 

13 

Source:  Joint  Lessons  Learned  Information  System. 

Note:  This  includes  active  records  on  the  Non-Secure  Internet  Protocol  Router  Network  as  of  April  9, 
2009;  and  on  the  Secure  Internet  Protocol  Router  Network  (classified  and  unclassified)  as  of  May  4, 
2009 

"Includes  unclassified  active  records  from  the  Secure  Internet  Protocol  Router  Network. 

The  philosophy  and  approach  of  NORTHCOM’s  Lessons  Learned  Program 
have  been  largely  the  same  since  NORTHCOM  published  its  first 
instruction  for  the  program  in  2003, 74  although  the  requirement  to  re¬ 
observe  corrective  actions  in  a  subsequent  exercise  or  operation  before 
closing  them  was  not  established  until  2005. 75  We  found  that  NORTHCOM 
generally  has  a  systematic  lessons  learned  and  corrective  action  program, 
based  on  clear  procedures  and  a  regular  process.  Observations  are 
assigned  to  an  office  of  primary  responsibility  within  NORTHCOM  and 
categorized  as  either  a  lesson  learned — a  positive  finding — or  an  issue 
which  requires  corrective  action.  NORTHCOM’s  intent  is  to  manage  and 
resolve  issues  requiring  corrective  action  at  the  lowest  organizational  level 
possible.  This  responsibility  is  generally  within  NORTHCOM’s  various 
directorates,  component  commands,  or  a  Joint  Task  Force.76  Issues  may 
be  closed  at  the  directorate  level  without  external  approval  or  oversight. 


,4U.S.  Northern  Command,  Lessons  Learned  Policy  and  Guidance,  Operating  Instruction 
36-104,  (Colorado  Springs,  Colo.:  Oct.  1,  2003). 

7BChairman  of  the  Joint  Chiefs  of  Staff,  Instruction  3150.26B,  Joint  Lessons  Learned 
Program,  A-2  -A-3  (Washington,  D.C.:  February  15,  2005). 

76Those  component  commands,  such  as  Army  Forces  North  or  Air  Force  North,  may  also 
have  a  lessons  learned  program  of  their  own. 
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Broader  scope  or  more  sensitive  issues  requiring  the  involvement  of  more 
than  one  directorate  or  subcommand  go  into  the  formal  corrective  action 
board  process  for  review,  tracking,  and  approval  as  necessary.77  This 
formal  process  includes  two  boards — the  Corrective  Action  Board  and  the 
Executive  Corrective  Action  Board — to  review  and  resolve  issues.78  Figure 
7  illustrates  the  flow  of  NORTHCOM’s  lessons  learned  and  corrective 
action  process. 


'  The  Corrective  Action  Board  process  is  intended  to  provide  a  means  for  addressing 
commandwide  issues  and  ensuring  these  issues  are  tracked,  resolved,  and  verified. 

78The  Corrective  Action  Board  Working  Group  is  chaired  by  the  head  of  the  Joint  Training 
and  Exercises  Directorate,  and  includes  representatives  from  each  NORTHCOM 
directorate  and  subcommand  and  reviews  issues  requiring  extensive  coordination  across 
directorates  and  subordinate  commands  or  action  by  an  organization(s)  external  to 
NORTHCOM.  The  Executive  Corrective  Action  Board  is  chaired  by  NORTHCOM’s  Chief  of 
Staff  and  comprised  of  the  heads  of  the  directorates  and  subcommands  and  reviews 
sensitive  issues  requiring  decisions  by  senior  leaders  or  actions  of  organizations  outside  of 
NORTHCOM,  such  as  the  Joint  Staff.  For  example,  lessons  learned  that  are  shared  with 
DHS  are  reviewed  by  the  Executive  Corrective  Action  Board  before  they  are  forwarded  to 
the  Joint  Staff. 
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Figure  7:  NORTHCOM  Lessons  Learned  and  Corrective  Action  Process 


Source:  GAO  analysis  of  NORTHCOM  guidance. 
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Gaps  Remain  with 
Collecting  Observations 
From  and  Sharing  Lessons 
Learned  with  Interagency 
Partners  and  States 


JLLIS  was  intended  to  make  lessons  learned  more  widely  available; 
however,  we  found  that  non-DOD  exercise  participants  have  limited 
access  to  this  system,  which  presents  challenges  for  NORTHCOM  in 
collecting  observations  from  and  sharing  lessons  learned  with  interagency 
partners  and  states.79  Joint  Staff  and  NORTHCOM  officials  told  us  that 
only  a  small  number  of  interagency  staff  and  no  state  representatives 
currently  have  access  to  JLLIS  and  therefore  are  not  able  to  directly  input 
exercise  observations  or  review  NORTHCOM’s  lessons  learned  through 
this  system.80  DOD  officials  told  us  that  access  to  JLLIS  is  restricted  in 
order  to  safeguard  sensitive  information  because  of  concerns  that 
adversaries  may  be  able  to  exploit  any  weaknesses  contained  in  those 
records.  On  the  other  hand,  NORTHCOM  officials  told  us  that  most  of  the 
observations  collected  from  exercises  are  unclassified,  and  we  found  this 
to  be  true  for  more  than  90  percent  of  the  observations  from 
NORTHCOM’s  six  most  recent  exercises.  The  vast  majority  of  the 
observations  we  reviewed  are  listed  as  unclassified  and  few  records 
indicate  they  have  a  “For  Official  Use  Only”  restriction.  For  users  to  access 
the  unclassified  portion  of  the  JLLIS  they  must  have  a  DOD  common 
access  card,  a  card  reader,  and  appropriate  computer  software  prior  to 
requesting  access.  Special  security  arrangements  must  be  made  for 
granting  JLLIS  access  to  civilian  federal  officials  who  do  not  possess 
common  access  cards  but  have  other  federally  issued  personal 
identification  verification  cards.  These  officials  cannot  access  JLLIS 
because  they  also  lack  the  appropriate  software  to  operate  the  personal 
identification  verification  cards  within  the  system.  In  an  attempt  to 
address  this  gap  and  gather  more  input  from  non-DOD  exercise 
participants,  in  April  2008  NORTHCOM  officials  placed  a  Word  document 
template  on  its  internet  portal  to  provide  a  venue  for  non-DOD  exercise 
participants  to  record  exercise  observations.  Any  observations  provided  in 
this  format  would  be  input  by  NORTHCOM  Joint  Training  and  Exercise 


'9NORTHCOM  has  a  process  for  sharing  lessons  learned  from  National  Level  Exercises  that 
are  applicable  to  the  interagency  to  DHS  through  the  Joint  Staff.  Thus  far,  NORTHCOM  has 
provided  three3  issues  requiring  corrective  action  from  National  Level  Exercise  2-08 
conducted  in  May  2008. 

'Joint  Staff  officials  told  us  that  about  lODepartment  of  State  officials  have  access  to 
JLLIS,  and  DHS  and  Department  of  Energy  officials  have  requested  access  for  some  of  their 
staff.  Other  departments  and  agencies,  such  as  the  Federal  Bureau  of  Investigation  and 
Department  of  Health  and  Human  Services,  have  received  briefings  about  JLLIS,  but  have 
not  requested  access.  The  Joint  Staff  official  also  told  us  that  his  office  does  not  have 
enough  staff  to  support  a  large  number  of  non-ommon  access  card  users  requesting  JLLIS 
access,  and  granting  access  would  be  a  lengthy  process  due  to  the  software  and  security 
requirements  that  must  be  addressed. 
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Directorate  officials  into  JLLIS.  However,  NORTHCOM’s  portal  requires 
the  same  DOD-issued  card  to  gain  entry.81  In  addition,  NORTHCOM’s 
lessons  learned  manager  told  us  no  one  has  submitted  observations  using 
the  template  since  it  was  put  on  the  portal.  This  may  be  because  the 
command  has  not  actively  publicized  how  to  access  the  template  and 
underscored  the  value  to  the  command  of  obtaining  observations  from 
interagency  partners  and  states.  In  response  to  our  inquiries  in  May  2009, 
NORTHCOM’s  lessons  learned  manager  told  us  that  the  command  is  in  the 
process  of  adding  a  link  to  DHS’s  Homeland  Security  Information  Network 
so  that  interagency  partners  and  states  will  be  able  to  submit  lessons 
learned  which  can  subsequently  be  transferred  to  JLLIS  by  NORTHCOM 
officials.82 

In  addition  to  collecting  observations  using  JLLIS,  NORTHCOM  can  obtain 
lessons  learned  from  interagency  partners  and  states  during  postexercise 
meetings.  NORTHCOM  conducts  a  review  called  a  Hotwash  within  hours 
of  completing  the  exercise  so  that  exercise  participants  can  discuss 
observations  that  significantly  impacted  their  mission  and  recommend 
emergent  themes  for  discussion  during  a  subsequent  review  known  as  the 
facilitated  after-action  review.  This  review,  generally  held  7  days  after  the 
exercise  is  completed,  provides  an  opportunity  to  present  major  issues  to 
senior  leaders  and  obtain  the  Commander’s  guidance  for  resolution. 
However,  the  extent  to  which  interagency  and  state  officials  are  attending 
and  participating  in  NORTHCOM’s  postexercise  meetings  is  unclear. 

Based  on  NORTHCOM’s  documentation,  only  two  states  (out  of  the  last 
six  major  exercises)  participated  in  a  Facilitated  After  Action  Review — 
California  in  Vigilant  Shield  09  and  Alaska  in  Ardent  Sentry  07.  Officials 
from  three  states  in  addition  to  California  told  us  that  they  participated  in 
the  after-action  meeting  for  the  exercises  they  participated  in,  but  they 
may  have  participated  in  the  regional  or  national-level  meeting  rather  than 
NORTHCOM’s.83  Officials  from  two  of  the  seven  states  we  met  with  told  us 
that  they  did  not  attend  NORTHCOM’s  postexercise  reviews  for  the 


slNORTHCOM’s  lessons  learned  manager  told  us  that  those  without  cards  that  have  a  valid 
need  for  access  to  the  portal  must  work  with  NORTHCOM’s  Help  Desk  staff  to  register  for 
an  account. 

82DHS’s  Web-based  Homeland  Security  Information  Network  facilitates  information  sharing 
and  collaboration  between  federal,  state,  local,  tribal,  private  sector,  and  international 
partners  involved  in  homeland  security. 

83Depending  on  the  exercise,  for  example  if  it  is  linked  to  a  national  level  or  regional 
exercise,  multiple  Facilitated  After  Action  Reviews  may  take  place. 
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exercises  in  which  they  participated  at  least  partly  due  to  staffing  and 
budget  limitations. 

NORTHCOM  has  also  attempted  to  share  lessons  learned  with  other 
federal  agencies  and  states  by  using  FEMA’s  lessons  learned  sharing 
system.  For  example,  NORTHCOM  has  posted  six  reports  onto  FEMA’s 
lessons  learned  system,  including  four  recent  exercise  reports  and  two 
reports  from  operations  in  2008. 84  However,  with  one  exception,  the 
documents  that  NORTHCOM  has  made  available  on  this  system  (1) 
include  only  lists  of  observations  and,  in  some  cases,  record-tracking 
numbers  from  JLLIS  and  previous  lessons  learned  systems,  and  (2)  lack 
detailed  information  on  individual  lessons  learned  and  corrective  actions.85 
Joint  Staff  and  NORTHCOM  officials  told  us  that  they  do  not  post  detailed 
information  on  the  unclassified  Lessons  Learned  Information  Sharing 
system  Web  site,  because  it  is  not  adequately  protected  from  the  potential 
for  unauthorized  access  to  records.  As  a  result,  the  security  of  the 
information  cannot  be  assured.  According  to  these  officials,  if  an 
adversary  nation  or  terrorist  group  gained  access  to  this  information,  it 
may  be  possible  for  them  to  identify  weaknesses  in  NORTHCOM’s 
operations  that  can  be  exploited.  In  a  recent  exercise  summary  report 
NORTHCOM  stated  that  it  will  post  lessons  learned,  best  practices,  and 
reports  that  may  benefit  their  non-DOD  mission  partners  in  FEMA’s 
Lessons  Learned  Information  Sharing  system,  which  the  report  describes 
as  a  secure,  restricted-access  information  system.86  Because  security 
concerns  are  preventing  NORTHCOM  from  openly  sharing  all  its 
unclassified  lessons  learned  with  its  interagency  partners  and  the  states, 
the  information  NORTHCOM  does  provide  may  be  of  limited  value  for 
helping  its  partners  improve  the  nation’s  disaster  responsiveness.  Because 
NORTHCOM  is  not  fully  involving  other  federal  agencies  and  states  in  its 
lessons  learned  process,  it  is  missing  opportunities  to  learn  lessons  from 
an  exercise.  For  example,  officials  from  two  states  did  not  provide 
NORTHCOM  with  lessons  learned  from  exercises  because  they  did  not 


84NORTHCOM  reports  posted  on  FEMA’s  Lessons  Learned  Information  Sharing  system 
include  exercise  summary  reports  for  National  Level  Exercise  1-08  (Vigilant  Shield  08),  and 
Ardent  Sentry  2007;  two  copies  of  the  Quick  Look  Report  for  National  Level  Exercise  2-08; 
NORTHCOM  Support  for  2008  Hurricane  Season  Operations  Summary  Report;  and 
NORTHCOM  Support  for  2008  Political  Conventions  Operations  Summary  Report. 

85We  found  one  document  where  details  were  provided  on  12  observations. 

86U.S.  Northern  Command,  National  Level  Exercise  2-08  Exercise  Summary  Report 
(Colorado  Springs,  Colo.:  Aug.  15,  2008),  16. 
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attend  the  command’s  post  exercise  reviews.  As  a  result,  NORTHCOM 
risks  the  reoccurrence  of  potential  problems  that  were  not  identified  in  its 
process. 


NORTHCOM  Faces 
Challenges  Managing 
Corrective  Actions 


DOD  and  NORTHCOM  guidance  requires  that  issues  requiring  corrective 
actions  be  tracked  and  remain  open  until  the  solutions  are  completed  and 
verified  as  effective — through  training,  operations,  or  exercises.87  We 
found  that  NORTHCOM  directorates  and  subcommands  are  closing  some 
issues  prematurely,  without  confirming  that  corrective  actions  were  made 
or  verifying  in  a  subsequent  exercise  or  operation  that  the  corrective 
action  is  effective.  We  reviewed  unclassified  records  in  JLLIS  from 
NORTHCOM’s  previous  six  large-scale  exercises  and  found  at  least  77  of 
the  375  records  or  about  20  percent  required  corrective  actions  but  were 
either  closed  prior  to  completing  the  corrective  action  or  closed  without 
verifying  the  effectiveness  of  the  corrective  action.88  For  example,  an 
observation  was  made  during  Ardent  Sentry  07  that  NORTHCOM  did  not 
have  a  process  for  addressing  a  foreign  nation’s  offer  of  military-to-military 
assistance  in  a  major  disaster.  The  issue  was  validated  and  the  corrective 
action  developed,  but  the  issue  was  closed  by  the  originating  organization 
before  the  corrective  action  could  be  verified  or  reobserved  in  a 
subsequent  exercise.89  The  record  was  closed  even  though  the  Executive 
Corrective  Action  Board  directed  that  it  remain  open  until  an  exercise  of 
suitable  scope  to  require  significant  military  support  was  developed.90 
Another  example  of  a  record  being  closed  without  verification  or  re¬ 
observation  is  an  observation  made  during  Ardent  Sentry  07  raising 
concerns  that  NORTHCOM  personnel  could  arrive  to  assess  a  disaster  site 


s‘Chairman  of  the  Joint  Chiefs  of  Staff,  Instruction  3150.25D,  Joint  Lessons  Learned 
Program  (Washington,  D.C.:  Oct.  10,  2008),  A-8;  U.S.  Northern  Command,  Lessons  Learned 
Program  and  Corrective  Action  Board  Process,  Instruction  16-166  (Colorado  Springs,  CO: 
Mar.  16,  2009),  14;  Prior  Chairman  of  the  Joint  Chiefs  of  Staff  guidance  also  required  that 
corrective  actions  be  tracked  and  remain  open  until  the  solutions  are  completed  and 
verified. 

88We  reviewed  active  records  from  NORTHCOM’s  large-scale  exercises  that  were 
conducted  between  May  2006  and  November  2008  for  Ardent  Sentry  2006,  2007,  and  2008 
and  Vigilant  Shield  2007,  2008,  and  2009. 

89The  corrective  action  included  a  set  of  checklists  and  processes  added  to  the  command’s 
battle  staff  operating  procedures. 

90The  scenarios  developed  for  NORTHCOM’s  next  two  applicable  major  exercises  were  not 
of  sufficient  scope  to  induce  another  nation,  other  than  Canada,  to  offer  military 
assistance. 
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without  alerting  state  officials  they  would  be  coming.  As  a  result, 
NORTHCOM  developed  a  new  Command  Assessment  Element  Concept  of 
Execution91  in  July  2007  to  promote  better  command  and  control  and 
situational  awareness;  however,  the  issue  was  closed  before  the  procedure 
could  be  observed  in  a  subsequent  exercise  or  operation  to  verify  effective 
resolution. 

These  issues  are  likely  being  closed  without  verification  or  re-observation, 
because  NORTHCOM  Training  and  Exercise  Directorate  officials  do  not 
have  oversight  over  the  disposition  of  open  issues  that  are  resolved  within 
directorates  or  are  unable  to  give  long-standing  issues  the  sustained 
management  attention  needed  to  ensure  resolution.  NORTHCOM’s  lessons 
learned  manager  told  us  that  the  command  does  not  have  the  staff 
necessary  to  oversee  the  actions  on  records  handled  within  the  other 
directorates.  In  addition,  while  the  checkbox  format  in  JLLIS  makes  it  easy 
to  see  whether  an  issue  is  open,  awaiting  verification,  or  closed,  entries 
made  in  JLLIS  regarding  corrective  actions  required,  implementation  date, 
and  plan  for  verification  are  primarily  in  a  narrative  format,  which  may 
make  the  review  and  oversight  process  more  time  consuming.  Without 
sufficient  oversight,  NORTHCOM  cannot  ensure  that  corrective  actions 
are  verified  and  reobserved  in  a  subsequent  exercise  or  operation  before 
the  issue  is  closed,  so  that  the  command  knows  the  solution  is  effective. 
We  recognize  that  such  oversight  should  be  addressed  without 
significantly  stressing  NORTHCOM’s  staff.  However,  if  NORTHCOM  does 
not  ensure  that  corrective  actions  are  fully  resolved,  it  increases  the  risk 
that  these  issues  may  occur  again,  possibly  during  crucial,  real-world 
situations.  This  lack  of  oversight,  coupled  with  the  lack  of  a  well-thought 
out  and  consistent  process  for  including  the  states  in  assessing  exercises 
as  discussed  earlier  in  this  report,  further  limits  the  knowledge  gained  and 
value  of  the  exercise  for  all  participants. 


91The  Command  Assessment  Element  is  a  team  supporting  NORTHCOM’s  commander  by 
providing  situational  awareness  from  a  disaster  or  incident  site. 
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NORTHCOM  Has 
Taken  Steps  to 
Integrate  Its 
Exercises  with  the 
National  Exercise 
Program,  but 
Guidance  Is  Not 
Consistently  Applied 

Since  the  NEP  Charter  was  approved  in  January  2007,  NORTHCOM  has 
participated  in  the  major  national  exercise  held  under  the  NEP  and  taken 
steps  to  integrate  its  exercises  into  the  national  program.92  NEP  guidance 
requires  that  heads  of  departments  and  agencies  actively  participate  in  tier 

1  exercises  and  recommends  participation  in  tier  II  exercises  either 
through  the  National  Exercise  Simulation  Center  or  as  determined  by 
agency  leadership.93  Departments  or  agencies  can  participate  in  the  NEP 
by  combining  an  existing  exercise  with  a  NEP  exercise,  taking  part  in  a 
tier  II  exercise  sponsored  by  a  different  department  or  agency,  or 
requesting  to  lead  a  tier  II  exercise  to  obtain  greater  interagency 
participation  and  support.  DOD  guidance  requires  that  components 
participate  in  or  lead  planning  efforts  of  NEP  exercises  as  appropriate 
given  the  scenario  or  as  tasked  by  the  Assistant  Secretary  of  Defense, 
Homeland  Defense  and  America’s  Security  Affairs  or  the  Chairman  of  the 
Joint  Chiefs  of  Staff.94  NORTHCOM’s  training  guidance  specifies  the  NEP 
exercises  in  which  the  command  plans  to  participate  during  the  following 

2  fiscal  years.95  NORTHCOM  combined  two  of  its  large-scale  exercises — 
Vigilant  Shield  08  and  Ardent  Sentry  08 — with  major  national  exercises 


NORTHCOM  Has 
Participated  in  National 
Exercises  and  Taken  Steps 
to  Integrate  with  the  NEP 


92For  the  purpose  of  this  report,  we  are  defining  major  national  exercises  as  operations- 
based  NEP  exercises  involving  the  deployment  of  personnel  or  forces  to  conduct 
operations  from  the  federal,  state,  and  local  levels. 

93DHS,  National  Exercise  Program  Implementation  Plan. 

94DOD,  DOD  Participation  in  the  National  Exercise  Program,  Instruction  3020.47, 
(Washington,  D.C.:  Jan.  29,  2009),  10. 

95The  command’s  Joint  Training  Plan,  which  includes  the  commander’s  training  guidance,  is 
revised  and  updated  annually.  U.S.  Northern  Command,  Commander’s  Training  Guidance 
for  FY  09-10. 
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and  has  taken  part  in  two  additional  exercises  sponsored  by  other 
departments  (see  table  6).  For  example,  National  Level  Exercise  1-08,  a 
tier  I  exercise,  and  NORTHCOM’s  Vigilant  Shield  08  were  conducted 
October  15  -20,  2007,  in  parallel  with  Top  Officials  496  and  several  other 
exercises.97  These  exercises  were  linked  together  by  the  use  of  common 
scenarios  and  objectives  intended  to  test  existing  plans,  policies,  and 
procedures  to  identify  planning  and  resource  gaps  and  develop  corrective 
actions  to  improve  preparedness  against  a  weapons  of  mass  destruction 
attack. 


Table  6:  NORTHCOM  Participation  in  National  Level  and  Tier  II  Exercises  Since  2007 

Exercise  and  tier 

Date 

Main  scenario 

Primary  sponsor/lead 

Description  of 

NORTHCOM 

participation 

National  Level 

Exercise  1-08  (1) 

October  2007 

Radiological  dispersal  device 
attack 

FEMA 

Linked  Vigilant  Shield  08 
to  exercise 

National  Level 

Exercise  2-08  (II) 

May  2008 

Hurricane,  terrorist  chemical 
attack,  accidental  chemical 
release 

FEMA 

Linked  Ardent  Sentry  08 
to  exercise 

Diablo  Bravo  (II) 

July  2008 

Nuclear  weapons  incident 

Department  of  Energy/ 
National  Nuclear  Security 
Administration 

Provided  initial  DOD 
response  support  for 
Department  of  Energy 

Empire  2009  (II) 

June  2009 

Radiological  dispersal  device 
Attack 

National  Nuclear  Security 
Administration 

Provided  Defense 
Coordinating  Officer 
support 

Source:  GAO  analysis  of  DHS  and  NORTHCOM  documentation. 


Note:  We  limited  the  scope  of  our  finding  to  major  national  exercises  sponsored  at  the  federal  level 
and  planned  after  National  Level  Exercise  1  -08,  the  first  major  exercise  conducted  under  the  NEP. 


NORTHCOM  officials  told  us  that  they  generally  would  like  to  participate 
in  NEP  exercises  to  achieve  the  benefits  of  exercising  with  interagency 
partners,  but  in  some  cases  it  is  not  beneficial  to  do  so.  For  example,  the 
officials  told  us  NORTHCOM  decided  not  to  combine  Ardent  Sentry  09 


96The  Top  Officials  exercise,  better  known  as  TOPOFF,  is  the  nation’s  terrorism 
preparedness  exercise  involving  officials  at  every  level  of  government  and  representatives 
from  the  international  community  and  private  sector.  Sponsored  by  DHS,  TOPOFF  4 
shared  the  same  radiological  dispersal  device  scenario  in  Vigilant  Shield  08  and  included 
more  than  100  organizations. 

9lNational  Level  Exercise  1-08  included  Global  Lightning  08  at  U.S.  Strategic  Command, 
Positive  Response  08-1  for  the  Chairman,  Joint  Chiefs  of  Staff,  Blue  Flame  in  the  United 
Kingdom,  Pacific  Challenge  in  Australia,  a  U.S.  Southern  Command  Response  Cell  due  to 
Able  Warrior  08-1  date  change,  and  an  unnamed  exercise  at  U.S.  Pacific  Command. 
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with  National  Level  Exercise  09 — a  tier  1  exercise  scheduled  for  July 
2009 — because  the  objectives  and  scenarios  for  the  exercises  did  not  meet 
their  training  needs.  Although  NORTHCOM  officials  will  conduct  Ardent 
Sentry  09  separately,  they  are  using  the  National  Exercise  Simulation 
Center — FEMA’s  newly  established  training  and  exercise  facility — to 
provide  a  test  run  for  the  center’s  use  in  National  Level  Exercise  09. 98 

DOD  and  NORTHCOM  have  taken  steps  to  integrate  exercises  with  the 
National  Exercise  Program,  including  posting  the  command’s  exercises  on 
DHS’s  National  Exercise  Schedule,  successfully  applying  to  lead  a  tier  II 
exercise,  and  publishing  guidance  on  integration  with  the  NEP.  The  NEP 
Implementation  Plan  recommends  that  federal  departments  and  agencies 
post  exercises  on  the  NEP’s  National  Exercise  Schedule  so  that  exercises 
and  planning  meetings  can  be  synchronized  across  the  federal 
government.99  NORTHCOM  has  posted  its  annual  Ardent  Sentry  and 
Vigilant  Shield  exercises  for  the  first  4  of  5  fiscal  years  on  the  national 
schedule,  while  FEMA’s  National  Exercise  Division  has  posted  exercises 
for  the  first  3  fiscal  years.  As  of  June  2009,  neither  the  Joint  Staff  nor  any 
other  combatant  commands  have  posted  exercises  on  the  national 
schedule.  In  addition,  NORTHCOM  recently  requested  and  was  granted 
approval  to  lead  Vigilant  Shield  10  as  a  tier  II  exercise  scheduled  for 
November  2009.  Vigilant  Shield  10  should  have  greater  interagency 
participation  than  it  would  have  received  as  a  tier  3  exercise,  since  federal 
departments  and  agencies  will  be  required,  at  a  minimum,  to  participate  in 
the  National  Exercise  Simulation  Center.  As  of  May  2009,  the  participants 
of  Vigilant  Shield  10  include  the  DHS  and  the  Departments  of  Justice, 
Energy,  Transportation,  Health  and  Human  Services,  and  Veteran  Affairs; 
the  Joint  Chiefs  of  Staff  and  U.S.  Joint  Forces  Command;  and  other 
government  and  nongovernment  organizations.100  This  exercise  will  be  the 
first  time  that  NORTHCOM  will  share  planning  responsibilities  with 
FEMA’s  National  Exercise  Division.  This  exercise  will  also  be  linked  to  a 


"FEMA’s  National  Exercise  Simulation  Center  opened  on  January  12,  2009,  to  provide  a 
national,  shared  facility  to  train  and  exercise  and  improve  coordination  for  all-hazards 
preparedness.  The  goal  was  to  pool  resources,  maximize  efficiency,  and  provide  sustained 
exercise  and  training  support  to  all  participants. 

"FEMA  officials  told  us  that  only  tier  1  and  2  exercises  are  required  to  be  included  in  the 
national  schedule,  but  encouraged  the  inclusion  of  Tier  3  exercises. 

100The  list  of  participating  departments  or  agencies  may  change  as  the  planning  for  Vigilant 
Shield  10  progresses. 
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Canadian  government  an  exercise  to  demonstrate  its  readiness  for  the 
2010  Olympics  in  Vancouver. 


NORTHCOM  Has  Not  NEP  guidance  includes  policies  and  tools  for  the  design,  planning, 

Consistently  Applied  NEP  conduct,  and  evaluation  of  exercises — known  as  the  Homeland  Security 

Guidance  Exercise  and  Evaluation  Program,  which  creates  a  common  exercise 

policy  and  consistent  terminology  for  exercise  planners  and  serves  as  the 
foundation  of  NEP  exercises.101  FEMA  requires  that  entities,  such  as  state 
and  local  governments,  receiving  homeland  security  grant  funding  for  their 
exercises  adhere  to  specific  Homeland  Security  Exercise  Evaluation 
Program  guidance  for  exercise  program  management,  design,  conduct, 
evaluation,  and  improvement  planning. 102  We  reviewed  key  program 
documents,  such  as  the  Implementation  Plan,  and  found  that  this  guidance 
is  unclear  about  the  extent  to  which  federal  agencies  should  use  the 
Homeland  Security  Exercise  Evaluation  Program.  For  example,  the 
Implementation  Plan  states  that  the  NEP  does  not  displace  a  preexisting 
exercise  program,  and  none  of  the  NEP  guidance  requires  that  federal 
agencies  use  the  Homeland  Security  Exercise  Evaluation  Program. 
However,  the  Implementation  Plan  states  that  the  Homeland  Security 
Exercise  Evaluation  Program  will  serve  as  the  doctrinal  foundation  for 
NEP  exercises. 

FEMA  officials  told  us  that  federal  agencies  should  use  this  program  when 
participating  in  tier  I  and  tier  II  exercises  so  that  the  various  exercise 
participants  have  consistency  when  planning,  conducting,  and  assessing 
exercises.  We  found  that  NORTHCOM  generally  has  used  DOD’s  Joint 
Training  System  guidance  for  planning  NEP  exercises,  defining 
capabilities,  and  reporting  exercise  results.103  NORTHCOM  officials  told  us 
that  the  Joint  Training  System  is  consistent  with  the  NEP  and  served,  in 
part,  as  the  basis  for  the  Homeland  Security  Exercise  and  Evaluation 


101This  plan  was  revised  in  June  2008,  but  has  not  been  formally  approved  by  the  current 
administration.  However,  FEMA  officials  told  us  that  they  intend  to  execute  the  NEP  using 
the  latest  revision  of  the  plan.  DHS,  DRAFT  National  Exercise  Program  Implementation 
Plan  (Washington,  D.C.:  June  20,  2008). 

102GAO-09-369,  p.  35. 

103NORTHCOM  officials  told  us  that  they  have  used  Homeland  Security  Exercise  Evaluation 
Program  guidance  in  one  case  when  they  drafted  the  exercise  plan  for  NLE  2-08,  because 
their  state  and  local  exercise  partners  were  required  to  use  this  guidance. 
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Program. 104  We  found  that  these  sets  of  guidance  have  similar  processes 
but  use  different  methods  for  defining  the  tasks  and  capabilities  that  are 
performed  and  validated  in  an  exercise.  The  primary  differences  between 
these  sets  of  guidance  are  that  (1)  DOD’s  task  list,  which  serves  as  the 
basis  for  its  exercises,  includes  tasks  that  are  specific  to  military  missions, 
such  as  troop  movements  and  sealifts;  (2)  DHS  guidance  provides  more 
detailed  criteria  for  the  postexercise  documentation,  such  as  content  and 
format;  and  (3)  DHS’s  planning  cycle  is  generally  shorter — 9  to  15  months 
versus  12  to  18  months  for  DOD.  (See  table  7.)  See  app.  II  for  a  more 
detailed  comparison. 


Table  7:  Comparison  of  Areas  of  Application  between  DOD’s  Joint  Training  System  and  DHS’s  Homeland  Security  Exercise 
Evaluation  Program 


Areas  of  application 

Joint  Training  System 

Homeland  Security  Exercise  Evaluation  Program 

Methods  for  defining  tasks 
and  capabilities 

•  Derived  from  Joint  Mission  Essential  Task 
Lists. 

•  Users  may  also  develop  tasks  using  the 
Universal  Joint  Task  List. 

•  Derived  from  Target  Capabilities  List  or 

Universal  Task  List. 

•  Users  may  also  develop  tasks  using  the 

Universal  Task  List. 

Reporting  requirements 

•  Issue  a  Quick  Look  Report  and  an 

Executive  Summary  Report. 

•  No  detailed  guidelines  on  content  for  either 
report. 

•  No  detailed  guidelines  on  content  found  in 
the  required  Quick  Look  or  Exercise 
Summary  Report. 

•  Issue  a  Quick  Look  Report  and  an  After  Action 
Report/Improvement  Plan. 

•  Provides  detailed  guidelines  on  content  for  both 
reports,  including  templates. 

•  Provides  detailed  guidelines  on  content  for  both 
the  required  Quick  Look  Report  and  the  required 
After  Action  Report/Improvement  Plan. 

Planning  cycle 

•  Uses  the  Joint  Event  Life  Cycle,  has  five 
stages:  design,  planning,  preparation, 
execution,  and  evaluation. 

•  Intended  to  be  flexible  and  can  be  modified 
to  any  type  of  event.  Can  span  a  period  of 
12  to  18  months  for  exercises. 

•  Uses  the  Homeland  Security  Exercise  Evaluation 
System’s  Exercise  Cycle,  employs  five  phases: 
foundation,  design  and  development,  conduct, 
evaluation,  and  improvement  planning. 

•  Intended  to  be  adaptable  to  any  type  of  exercise. 
Can  span  a  period  of  9  to  15  months. 

Lessons  learned/corrective 
actions  systems 

•  Joint  Lessons  Learned  Information  System 
manages  and  tracks  Lessons  Learned  and 
corrective  actions. 

•  FEMA’s  Lessons  Learned  Information  Sharing 
System  manages  and  disseminates  lessons 
learned. 

•  DHS's  Corrective  Action  Program  system 
manages  corrective  actions. 


Source:  GAO  analysis  of  DOD  and  DHS  documentation. 


104NORTHCOM’s  training  guidance  for  fiscal  years  2008-2009  stated  that  the  command  will 
adhere  to  the  guiding  principles  of  the  Homeland  Security  and  Exercise  Evaluation 
Program,  but  this  reference  has  been  removed  from  guidance  for  fiscal  years  2009-2010. 
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According  to  both  the  Homeland  Security  Exercise  and  Evaluation 
Program  and  Joint  Training  System  guidance,  it  is  important  to  link  tasks 
and  capabilities  with  exercise  objectives  to  ensure  that  participants 
exercise  or  train  as  they  would  perform  in  a  real-world  event.  The 
Homeland  Security  Exercise  and  Evaluation  Program  recommends  using 
DHS’s  Target  Capabilities  List  or  the  Universal  Task  List  to  formulate  the 
tasks  and  capabilities  that  underlie  the  objectives  for  an  exercise.105  These 
lists  describe  the  capabilities  government  entities  need  and  tasks  they  are 
expected  to  perform  to  prevent,  protect  against,  respond  to,  and  recover 
from  incidents  of  national  significance.  In  contrast,  NORTHCOM  derives 
its  tasks  and  capabilities  from  the  Universal  Joint  Task  List  to  formulate 
Joint  Mission  Essential  Tasks.  According  to  NORTHCOM  guidance,  the 
command  is  required  to  include  in  its  exercises  the  Joint  Mission  Essential 
Tasks  associated  with  its  Joint  Training  Plan,  which  is  updated  annually. 106 
These  tasks  are  identified  by  joint  force  commanders  as  most  essential  to 
their  assigned  or  anticipated  missions  with  priority  given  to  their  wartime 
missions  and  describe  their  priority  wartime  requirements.  We  found  that 
DOD’s  operating  instruction  for  participation  in  the  NEP  does  not  provide 
guidance  on  how  DOD  components  should  incorporate  tasks  and 
capabilities  derived  from  sources  recommended  by  the  Homeland  Security 
Exercise  and  Evaluation  Program  when  participating  in  NEP  exercises. 107 
The  primary  differences  between  DHS’  and  DOD’s  lists  are  that  DOD’s  task 
lists  generally  incorporate  more  descriptive  metrics  and  criteria  to  assess 
performance  and  include  tasks  that  are  specific  to  military  missions,  such 
as  troop  movements  and  sealifts.  In  some  cases,  state  National  Guards 
officials  have  had  to  translate  DOD  task  lists  into  DHS  tasks  lists  when 
working  with  their  civilian  partners  and  vice  versa. 

We  also  found  that  neither  DOD’s  nor  NORTHCOM’s  guidance  for 
developing  postexercise  reports  includes  the  same  degree  of  specificity 
recommended  in  the  Homeland  Security  Exercise  Evaluation  Program. 

For  example,  both  sets  of  exercise  guidance  require  postexercise  reports; 
however,  the  Homeland  Security  Exercise  Evaluation  Program  provides 
templates  and  guidance  for  these  documents,  including  requiring  an 
improvement  plan  to  clearly  outline  the  corrective  actions  needed,  which 


10oDHS,  Homeland  Security  Exercise  and  Evaluation  Program,  Volume  1:  HSEEP 
Overview  and  Exercise  Program  Management  (Washington,  D.C.:  February  2007). 

106U.S.  Northern  Command,  Operations:  Exercise  Program,  pp.  4-5. 

10'DOD,  DOD  Participation  in  the  National  Exercise  Program,  Instruction  3020.47,  7-8. 
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are  not  included  in  DOD’s  or  NORTHCOM’s  guidance.  In  addition, 
NORTHCOM’s  exercise  summary  reports  for  National  Level  Exercise  1-08 
and  2-08  did  not  contain  all  information  recommended  by  the  Homeland 
Security  Exercise  Evaluation  Program.  For  example,  NORTHCOM  did  not 
include  the  recommended  analyses  regarding  the  capabilities  and  tasks 
tied  to  the  exercises’  objectives. 

As  stated  above,  we  reviewed  NEP  guidance  such  as  the  Implementation 
Plan  and  found  it  does  not  clearly  state  the  extent  to  which  federal 
agencies  are  required  to  follow  the  Homeland  Security  Exercise  and 
Evaluation  Program.  As  a  result  of  this  unclear  guidance,  we  found  that 
agency  officials  have  varying  interpretations  of  the  requirements.  For 
example,  a  DOD  and  a  Joint  Staff  official  told  us  that  NEP  guidance  does 
not  require  agencies  to  use  the  Homeland  Security  Exercise  Evaluation 
Program  even  for  NEP  exercises.  Therefore,  NORTHCOM  uses  the  Joint 
Training  System  rather  than  the  Homeland  Security  Exercise  and 
Evaluation  Program  as  the  basis  for  planning,  conducting,  and  assessing 
exercises.  However,  officials  from  FEMA’s  National  Exercise  Division  told 
us  that  all  participating  agencies  should  use  the  Homeland  Security 
Exercise  and  Evaluation  Program  guidance  for  tier  I  and  tier  II  NEP 
exercises. 108  FEMA  officials  stated  that  federal  departments  and  agencies 
should  be  held  accountable  for  meeting  key  requirements,  but  that  FEMA’s 
authority  is  limited  to  guiding,  supporting,  and  coordinating  with,  but  not 
directing  other  federal  departments  and  agencies  to  comply  with  guidance. 
As  we  have  previously  reported,  we  believe  that  FEMA’s  expanded 
leadership  role  under  the  Post-Katrina  Act  provides  FEMA  opportunities 
to  instill  a  shared  sense  of  responsibility  and  accountability  on  the  part  of 
all  agencies.109 

Neither  DOD  nor  NORTHCOM  guidance  specifically  addresses  the  extent 
to  which  DHS’s  Homeland  Security  Exercise  and  Evaluation  Program 
planning  and  documentation  requirements  should  be  followed.110  We 
recognize  that  NORTHCOM  and  DOD  must  meet  their  own  mission  and 


10SAccording  to  FEMA  officials,  since  tier  I  and  II  exercises  are  interagency  exercises,  the 
Homeland  Security  Exercise  and  Evaluation  Program  should  be  used  because  it  was 
designed  for  the  interagency  community. 

109GAO-09-369. 

110NORTHCOM  guidance  states  the  Vigilant  Guard  Program  should  use  Homeland  Security 
Exercise  and  Evaluation  Program  formats  to  encourage  civilian  agency  participation.  U.S. 
Northern  Command,  Commander’s  Training  Guidance:  Fiscal  Year  2009-2010,  A-18. 
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exercise  requirements  and  the  Joint  Training  System  may  be  best  suited 
for  NORTHCOM’s  exercises;  however,  all  of  the  states  we  visited  use 
Homeland  Security  Exercise  and  Evaluation  Program  guidance.  We  found 
that  having  differing  sets  of  guidance,  such  as  DOD’s  and  DHS’  capabilities 
task  lists  and  postexercise  documentation  requirements,  makes  exercises 
more  difficult  and  potentially  limits  the  benefits  for  participating  states. 

For  example,  officials  from  three  states  we  visited  told  us  that  using 
NORTHCOM’s  exercise  planning  and  reporting  requirements  rather  than 
Homeland  Security  Exercise  Evaluation  Program  guidance  has  made  the 
processes  more  difficult.  Further,  the  Defense  Science  Board  found  that 
inconsistent  approaches  to  the  development  and  content  of  postexercise 
documentation  may  affect  the  ability  of  organizations  to  fully  learn  lessons 
identified  in  exercises. 111  We  also  reported  that  when  other  federal  entities 
carry  out  processes  that  do  not  specifically  follow  the  Homeland  Security 
Exercise  and  Evaluation  Program,  FEMA  managers  do  not  have  the 
necessary  data  to  measure  progress,  identify  gaps  in  preparedness,  and 
track  lessons  learned — key  objectives  of  the  NEP.112  We  believe  that 
achieving  national  preparedness  requires  a  whole-of-government  approach 
and  is  a  shared  responsibility  among  federal,  state,  local,  and  tribal 
governments  and  organizations  and  an  integration  of  their  various 
standards,  policies,  and  procedures  into  the  national  system. 113 

Conclusions 

There  is  an  increasing  realization  within  the  federal  government  that  an 
effective,  seamless  national  response  to  an  incident  requires  a  strong 
partnership  among  federal,  state,  and  local  governments  and 
organizations,  including  integrated  planning,  training,  and  the  exercise  of 
those  plans.  For  DOD,  the  effective  execution  of  civil  support,  especially 
amid  simultaneous,  multijurisdictional  disasters,  requires  ever-closer 
working  relationships  with  other  departments  and  agencies  and  at  all 
levels  of  government.  NORTHCOM’s  use  of  DOD’s  Joint  Training  System 
has  provided  a  robust  process  for  planning  and  conducting  exercises  to 
improve  preparedness  to  achieve  its  homeland  defense  and  civil  support 
missions,  and  its  efforts  to  involve  its  interagency  partners  and  the  states 
in  exercises  have  helped  to  reduce  uncertainty  about  the  process  for 
responding  to  an  incident.  However,  without  a  consistent  record  of  what 

mDefense  Science  Board,  Unconventional  Operational  Concepts  and  the  Homeland. 

llzGAO-09-369,  p.  42. 

113GAO-09-369,  p.  68. 
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has  occurred  during  an  exercise  that  is  accessible  by  all  exercise 
participants,  including  those  from  other  federal  agencies  and  states, 
NORTHCOM  cannot  ensure  that  it  has  met  internal  standards,  trained  to 
key  focus  areas,  or  compared  the  goals  and  results  of  exercises  over  time. 
Further,  a  key  element  to  developing  effective  working  relationships  with 
all  states  is  a  consistent  process  for  including  states  in  planning  and 
executing  NORTHCOM’s  exercises  that  incorporates  state-specific 
knowledge  and  information.  By  coordinating  consistently  with 
organizations,  like  FEMA  and  NGB,  that  have  knowledge  and  experience 
dealing  with  states,  NORTHCOM  can  improve  the  value  and  effectiveness 
of  exercises  for  all  of  the  participants  involved. 

Exercises  provide  an  opportunity  to  enhance  preparedness  by  collecting, 
developing,  implementing,  and  disseminating  lessons  learned  and  verifying 
corrective  action  taken  to  resolve  previously  identified  issues. 
NORTHCOM’s  clear  procedure  for  capturing  observations  in  JLLIS  and 
identifying  issues  needing  corrective  action  has  helped  to  improve  its 
capabilities  to  complete  its  missions.  However,  by  not  providing  federal 
agencies  and  states  greater  access  to  its  lessons  learned  process, 
NORTHCOM  will  lose  opportunities  to  learn  valuable  lessons  from  an 
exercise,  particularly  observations  from  the  states  that  could  enhance 
coordination  and  build  more  effective  interagency  relationships.  Further, 
the  risk  that  issues  may  reoccur  will  be  increased,  particularly  when 
interagency  partners  are  not  aware  of  key  issues  or  concerns  that  might 
impede  the  government’s  overall  responsiveness  to  a  natural  or  man-made 
disaster.  In  addition,  when  corrective  actions  remain  open  until  fully 
implemented  and  verified  in  a  subsequent  exercise,  NORTHCOM  will  have 
greater  assurance  that  issues  raised  during  exercises  are  being  adequately 
addressed  and  the  corrections  are  in  fact  solving  the  problems  identified. 

NEP  policies  and  tools  for  the  design,  planning,  conduct,  and  evaluation  of 
exercises  are  intended  to  create  a  common  exercise  policy  and  consistent 
terminology  for  exercise  planners  across  all  levels  of  government  to 
improve  the  federal  government’s  ability  to  evaluate  national 
preparedness.  The  steps  DOD  and  NORTHCOM  have  taken  to  integrate 
exercises  with  the  NEP  have  helped  DHS  to  prioritize  and  coordinate 
federal  exercise  activities  and  enhance  the  federal  government’s  ability  to 
respond  to  an  incident.  We  recognize  that  NORTHCOM  and  DOD  must 
meet  their  own  mission  and  exercise  requirements  and  the  Joint  Training 
System  may  be  best  suited  to  meet  the  high  standards  required  for 
NORTHCOM’s  exercises.  However,  achieving  national  preparedness 
requires  shared  responsibility  among  federal,  state,  and  local  governments 
and  organizations  and  an  integration  of  their  various  standards,  policies, 


Page  59 


GAO-09-849  Homeland  Defense 


and  procedures  into  the  national  system.  We  also  recognize  that  the  NEP 
continues  to  evolve  and  become  more  useful  to  federal  and  state  partners. 
IHowever,  in  the  absence  of  clear  guidance  from  DHS  on  the  extent  to 
which  agencies  should  use  Homeland  Security  Exercise  Evaluation 
Program  planning  and  documentation  guidance,  DOD  should  ensure  that 
its  components  clearly  understand  when  the  use  of  this  guidance  is 
appropriate  so  that  both  DOD  and  its  exercise  partners,  such  as  other 
federal  agencies  and  states,  derive  the  most  benefits  from  exercises.  This, 
in  turn,  contributes  to  the  ultimate  success  of  a  whole-of-government 
approach  to  national  preparedness. 


Recommendations 
Executive  Action 


£q  To  improve  NORTHCOM’s  consistency  with  exercise  documentation,  we 
recommend  that  the  Secretary  of  Defense  direct  NORTHCOM’s 
Commander  to  develop  guidance  with  specific  criteria  for  postexercise 
documentation,  particularly  the  Exercise  Summary  Report  as  the  official 
exercise  record,  including  the  content  and  format  to  be  included  in  such 
reports  that  will  allow  the  results  and  lessons  learned  of  exercises  to  be 
easily  reviewed  and  compared. 


To  improve  NORTHCOM’s  involvement  of  interagency  partners  and  states 
in  its  exercises,  we  recommend  that 


•  the  Secretary  of  Defense,  in  consultation  with  the  Chairman  of  the 
Joint  Chiefs  of  Staff,  the  Commander,  U.S.  Northern  Command,  and 
other  relevant  combatant  commanders,  coordinate  with  the 
Department  of  Homeland  Security  and  Federal  Emergency 
Management  Agency  to  develop  guidance  and  procedures  for 
consistently  involving  state  officials  in  planning,  executing,  and 
assessing  exercises  that  incorporate  relevant  state-specific 
information,  and 

•  the  Secretary  of  Defense  direct  NORTHCOM’s  Commander  to  develop 
a  training  plan  for  NORTHCOM  headquarters  staff  on  state  emergency 
management  structures  and  relevant  issues  related  to  working  with 
civilian  state  and  local  emergency  management  officials. 

To  improve  NORTHCOM’s  involvement  of  interagency  partners  and  states 
in  its  lessons  learned  and  corrective  action  process  and  its  management  of 
corrective  actions,  we  recommend  that  the  Secretary  of  Defense  direct: 

•  NORTHCOM’s  Commander  to  establish  and  publicize  valid  and  easily 
accessible  procedures  for  non-DOD  exercise  participants  to  submit 
observations  relevant  to  NORTHCOM,  such  as  placing  a  template  on 
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NORTHCOM’s  publicly  accessible  Web  site  or  DHS’s  Homeland 
Security  Information  Network,  so  that  NORTHCOM  officials  have  a 
clear,  secure  avenue  to  obtain  observations  and  assess  potential 
lessons  that  originate  with  its  exercise  partners; 

•  the  Chairman  of  the  Joint  Chiefs  of  Staff,  in  consultation  and 
coordination  with  DHS,  to  either  resolve  information  assurance  issues 
so  that  the  combatant  commands,  including  NORTHCOM,  can  post 
Exercise  Summary  Reports  with  lessons  learned  and  observations 
from  NEP  exercises  on  DHS’s  Lessons  Learned  Information  Sharing 
system  to  make  them  easily  accessible  to  interagency  partners  and 
states  or  establish  an  alternative  method  to  systematically  collect  and 
share  lessons  learned;  and 

•  the  Chairman  of  the  Joint  Chiefs  of  Staff  to  revise  the  joint  lessons 
learned  operating  instruction  to  include  procedures  to  ensure  that 
appropriate  corrective  actions  are  implemented  and  verified  in  a 
subsequent  exercise  or  operation  before  being  closed  and  that  the 
reasons  for  closure  are  documented.  Possible  procedures  might  be 
adding  a  verification  checkbox  on  JLLIS’s  issue  management  page  or 
requiring  that  the  directorates  and  subordinate  commands  within  the 
combatant  commands  provide  a  status  report  when  a  correction  is 
implemented  and  reobserved  or  closed  for  reasons  other  than  re¬ 
observation. 

To  improve  NORTHCOM’s  ability  to  work  with  interagency  partners  on 
major  national  exercises  and  further  achieve  the  objectives  of  the  NEP,  we 
recommend  that  the  Secretary  of  Defense  revise  the  instruction  on  DOD 
participation  in  the  NEP  and/or  direct  the  Chairman  of  the  Joint  Chiefs  of 
Staff  to  revise  the  operating  instruction  regarding  DOD  participation  in  the 
NEP  to  provide  the  general  conditions  under  which  the  combatant 
commands  are  expected  to  follow  the  Homeland  Security  Exercise  and 
Evaluation  Program  planning  and  documentation  requirements  or  the 
DOD’s  Joint  Training  System  should  be  modified  for  those  civil  support 
exercises. 


Agency  Comments 
and  Our  Evaluation 


In  comments  on  a  draft  of  this  report,  DOD  generally  agreed  with  the 
intent  of  our  recommendations  and  discussed  steps  it  is  taking  or  plans  to 
take  to  address  these  recommendations.  DOD  also  provided  technical 
comments,  which  we  have  incorporated  into  the  report  where  appropriate. 
DHS  also  reviewed  a  draft  of  this  report  and  provided  technical  comments, 
which  we  have  incorporated  where  appropriate. 
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In  response  to  our  recommendation  that  NORTHCOM  develop  guidance 
with  specific  criteria  for  postexercise  documentation  to  allow  the  results 
and  lessons  learned  of  exercises  to  be  reviewed  and  compared,  DOD 
agreed  that  such  information  should  be  provided  in  a  standardized  format 
that  can  be  easily  accessed  and  understood  by  authorized  organizations 
which  might  benefit  from  such  knowledge.  DOD  cautioned  that  any 
actions  in  response  to  this  recommendation  must  accommodate 
constraints  regarding  classified  information.  We  agree  that  properly 
securing  classified  information  is  a  critical  responsibility  and  believe  this 
can  easily  be  accomplished  without  undermining  the  intent  of  the 
recommendation,  which  is  to  improve  the  consistency  and  completeness 
of  formal  exercise  documentation  and  thereby  its  overall  value. 

In  response  to  our  recommendation  that  DOD  coordinate  with  DHS  and 
FEMA  to  develop  guidance  and  procedures  for  consistently  involving  state 
officials  in  planning,  executing,  and  assessing  exercises  that  incorporate 
relevant  state-specific  information,  DOD  agreed  that  better  coordination 
for  interfacing  with  state  officials  can  be  achieved.  DOD  also  pointed  out 
that  NORTHCOM  continues  to  expand  its  efforts  to  work  through  defense 
coordinating  officers,  existing  state  National  Guard  relationships,  and 
FEMA  regional  headquarters  partners  to  ensure  that  states  are  able  to 
benefit  from  participation  in  DOD-sponsored  exercises.  However,  DOD 
also  said  that  while  NORTHCOM  has  continuously  engaged  and 
encouraged  state  participation  in  NORTHCOM-sponsored  exercises,  the 
primary  audience  for  such  training  is  and  must  remain  NORTHCOM.  DOD 
also  suggested  that  our  recommendation  has  applicability  to  other  federal 
interagency  partners  and  that  the  issue  should  be  addressed  to  the 
Exercise  and  Evaluation  Sub-Interagency  Planning  Committee  as  a 
revision  to  the  National  Exercise  Program  Implementation  Plan.  As  our 
report  indicates,  we  agree  that  NORTHCOM  has  sought  to  engage  and 
involve  the  states  in  its  comprehensive  exercise  program.  NORTHCOM 
plans  for  and  conducts  major  exercises  both  inside  and  outside  the 
construct  of  the  National  Exercise  Program.  Particularly  for  NORTHCOM- 
sponsored  exercises  focused  on  the  command’s  civil  support  mission,  the 
effective  involvement  of  and  interaction  with  state  and  other  federal 
partners  is  a  critical  component  of  improving  and  maintaining 
NORTHCOM’s  preparedness.  For  NORTHCOM’s  participation  in  national- 
level  exercises,  the  preparedness  goals  and  objectives  of  all  participants 
are  equally  important.  We  believe  that  in  developing  procedures  to 
improve  coordination  with  the  states,  DOD  can  (1)  avoid  situations  where 
exercises  meant  to  improve  preparedness  are  not  fully  coordinated  with 
the  necessary  partners;  (2)  capitalize  on  the  structures  and  organizations  it 
already  has  in  place,  such  as  the  defense  coordinating  officers  and 
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relationships  with  state  National  Guard  headquarters;  and  (3)  coordinate 
with  DHS  and  FEMA  to  improve  the  military-civilian  interface.  With  regard 
to  the  latter,  the  Exercise  and  Evaluation  Sub-Interagency  Planning 
Committee  may  indeed  be  one  of  the  venues  at  which  DOD  can  effectively 
coordinate  with  its  interagency  partners. 

With  respect  to  our  recommendation  that  NORTHCOM  develop  a  training 
plan  for  NORTHCOM  headquarters  staff  on  state  emergency  management 
structures  and  relevant  issues  related  to  working  with  civilian  state  and 
local  emergency  management  officials,  DOD  agreed  and  noted  that 
headquarters  training  is  required  for  all  newly  assigned  NORTHCOM  staff. 
Further,  DOD  noted  that  NORTHCOM  sponsors  three  versions  of  its 
defense  support  of  civil  authorities  seminar  that  are  targeted  to  staff  at 
different  seniority  levels.  We  agree  that  NORTHCOM  has  continued  to 
improve  the  level  of  awareness  and  training  it  provides  staff  on  the 
complexities  of  providing  defense  support  to  civilian  authorities  in  the 
United  States.  However,  this  does  not  fully  address  our  recommendation. 
While  training  on  the  general  procedures  of  the  national  response 
framework,  the  nature  of  state-federal  government  relations,  and  DOD’s 
proper  role  in  providing  military  support  to  civil  authorities  is  invaluable 
for  NORTHCOM  staff,  we  continue  to  believe  that  this  should  be 
supplemented  by  the  kinds  of  state-specific  information  that  would 
provide  both  exercise  officials  and  all  other  staff  with  an  understanding  of 
the  key  differences  between  states.  These  differences  are  possibly  as 
numerous  as  the  number  of  states  and  play  a  role  in  all  routine 
interactions  between  the  individual  states  and  DOD  officials  as  well  as  for 
effective  coordination  for  exercise  planning  and  coordination  during  a 
natural  disaster  or  some  other  no-notice  incident  requiring  defense 
support  to  civil  authorities. 

DOD  agreed  with  our  recommendation  that  NORTHCOM  establish  and 
publicize  valid  and  easily  accessible  procedures  for  non-DOD  exercise 
participants  to  submit  observations  relevant  to  NORTHCOM,  such  as 
placing  a  template  on  NORTHCOM’s  Web  site  or  DHS’  Homeland  Security 
Information  Network,  so  that  NORTHCOM  officials  have  a  clear,  secure 
avenue  to  obtain  observations  and  assess  potential  lessons  that  originate 
with  its  exercise  partners.  DOD  indicated  that  collecting  exercise 
information  from  all  perspectives  would  provide  additional  opportunities 
to  improve  NORTHCOM’s  ability  to  accomplish  its  mission  tasks. 

DOD  also  agreed  with  our  recommendation  that  it  work  with  DHS  to 
either  resolve  information  assurance  issues  so  that  NORTHCOM  can  post 
Exercise  Summary  Reports  with  lessons  learned  on  DHS’  Lessons  Learned 
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Information  Sharing  system  or  establish  an  alternative  method  to 
systematically  collect  and  share  lessons  learned.  DOD  cautioned  that 
while  wide  dissemination  of  information  approved  for  release  would  be  of 
great  benefit  to  homeland  security  entities  it  continues  to  adhere  to  the 
Joint  Training  System  and  cannot  mandate  that  DHS  alter  its  Lessons 
Learned  Information  System  to  make  accommodations.  DOD  also  noted 
that  it  has  procedures  in  place  to  allow  specifically  cleared  individuals 
from  outside  DOD  access  to  information  contained  in  Exercise  Summary 
Reports.  We  agree  that  DOD  cannot  mandate  alterations  to  the  Lessons 
Learned  Information  System.  We  also  agree  that  the  Joint  Training  System 
should  remain  the  chief  guidance  for  the  conduct  of  DOD  exercises. 
However,  we  continue  to  believe  that  in  working  with  DHS  on  the  proper 
level  and  mode  of  information  sharing,  DOD  may  be  able  to  improve  the 
dissemination  of  relevant  exercise-related  information  to  all  appropriate 
officials. 

DOD  agreed  with  our  recommendation  that  the  Chairman  of  the  Joint 
Chiefs  of  Staff  revise  the  joint  lessons  learned  operating  instruction  to 
include  procedures  to  ensure  that  appropriate  corrective  actions  are 
implemented  and  verified  in  a  subsequent  exercise  or  operation  before 
being  closed  and  that  the  reasons  for  closure  are  documented.  DOD 
indicated  that  the  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 
3150.25D  could  be  expanded  to  provide  more  guidance  and  the  Joint 
Lessons  Learned  Information  System  could  be  updated  to  provide  a 
technological  solution  to  address  the  issue  once  the  process  and 
procedures  are  in  place.  DOD  also  indicated  that  the  process  of  verifying 
corrective  action  and  closing  issues  will  become  more  effective  with  the 
modifications  it  outlined  in  response  to  the  recommendation. 

In  response  to  our  recommendation  that  DOD  revise  guidance  on  DOD 
participation  in  the  National  Exercise  Program  to  provide  the  general 
conditions  under  which  the  combatant  commands  are  expected  to  follow 
the  Homeland  Security  Exercise  and  Evaluation  Program  planning  and 
documentation  requirements  or  the  DOD’s  Joint  Training  System  should 
be  modified  for  those  civil  support  exercises,  DOD  recognized  the 
importance  of  ensuring  effective  interaction  with  interagency  partners  for 
homeland  security-related  exercises.  However,  DOD  noted  that  the 
National  Exercise  Program  Implementation  Plan  contains  language  placed 
there  at  DOD’s  insistence  that  establishes  a  process  to  resolve  doctrinal 
differences  during  exercise  planning.  DOD  indicated  that  together  with 
provisions  in  the  implementation  plan  establishing  the  administration, 
scope,  and  hierarchy  of  multiagency  homeland  security  exercises  and  the 
5-year  National  Exercise  Program  schedule,  this  should  address  our 
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recommendation.  DOD  further  noted  that  The  Joint  Training  System 
remains  the  Secretary  of  Defense’s  guidance  on  DOD  exercises  and  that 
the  National  Exercise  Program  Implementation  Plan  stipulates  that 
individual  department  or  agency  exercise  programs  should  not  be 
replaced.  We  agree  that  the  Joint  Training  System  is  and  should  be  DOD’s 
primary  guidance  for  ensuring  that  DOD  components  train  and  exercise 
according  to  standards.  However,  because  interagency  exercises  are 
becoming  an  ever  larger  part  of  the  national  preparedness  effort,  and  to 
the  extent  that  effective  exercise  planning  is  bolstered  by  common 
procedures,  our  recommendation  is  intended  to  help  DOD  clarify  for  its 
components  the  circumstances  under  which  the  specific  planning  and 
documentation  requirements  for  the  Homeland  Security  Exercise  and 
Evaluation  Program  can  be  followed  without  detriment  to  DOD’s  high 
training  and  exercise  standards  or  compromise  of  the  Joint  Training 
System. 

DOD’s  written  comments  are  reprinted  in  appendix  III. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense,  Secretary 
of  Homeland  Security,  and  other  interested  parties.  In  addition,  the  report 
will  be  available  at  no  charge  on  GAO’s  Web  site  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me 
at  (202)  512-5431  or  dagostinod@gao.gov.  Contacts  points  for  our  Office  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  Key  contributors  to  this  report  are  listed  in  appendix  IV. 


Davi  M.  D’Agostino 
Director,  Defense  Capabilities 
and  Management 
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Appendix  I:  Scope  and  Methodology 


In  conducting  this  review,  we  generally  focused  our  scope  on  U.S. 

Northern  Command’s  (NORTHCOM)  large-scale  exercises1  conducted 
since  Hurricane  Katrina  made  landfall  in  August  2005.  To  determine  the 
extent  to  which  NORTHCOM’s  exercise  program  is  consistent  with 
Department  of  Defense  (DOD)  training  and  exercise  requirements  and 
includes  relevant  exercise  partners,  we  evaluated  NORTHCOM’s 
compliance  with  exercise  reporting  and  documentation  requirements 
established  in  DOD  and  command  guidance.  We  reviewed  available 
guidance  to  determine  requirements  for  timing,  format,  and  content.  We 
also  compared  these  requirements  with  guidance  contained  in  the 
Department  of  Homeland  Security’s  (DHS)  Homeland  Security  Exercise 
and  Evaluation  Program  documentation.  We  reviewed  exercise 
documentation  for  all  large-scale  exercises  the  command  performed  since 
it  was  established  in  2002  to  determine  the  extent  to  which  the  command 
complied  with  the  guidance. 

We  also  performed  an  assessment  of  the  experiences  and  level  of 
participation  from  some  interagency  organizations  and  states  in 
NORTHCOM’s  large-scale  exercises.  We  initially  met  with  Nevada  officials 
who  participated  in  a  NORTHCOM  exercise  prior  to  Hurricane  Katrina — 
Determined  Promise  03 — to  provide  context  to  the  extent  that  changes 
may  have  been  made  to  NORTHCOM’s  exercise  program  and  help  develop 
our  state  selection  methodology.  We  selected  a  nongeneralizable  sample  of 
six  states  based  on  the  extent  to  which  they  have  participated  in  major 
NORTHCOM  exercise  since  Hurricane  Katrina  and  the  varying  scenarios 
of  the  exercises.  The  states  we  selected  played  a  major  role  in 
NORTHCOM  exercises  by  having  a  portion  of  the  exercise  conducted  in 
their  state  and  having  various  state  agencies  and  officials  participate. 

States  we  selected  include  Arizona,  California,  Michigan,  Oregon,  Rhode 
Island,  and  Washington.  We  met  with  representatives  from  each  state’s 
emergency  management  organization  and  state  national  guard.  Because  of 
the  methodology  selected,  the  resulting  data  and  information  from  these 
state  visits  could  not  be  projected  to  make  assumptions  about  the  rest  of 
the  states  and  what  they  may  experience  exercising  with  NORTHCOM.  We 
also  met  with  officials  from  three  Federal  Emergency  Management  Agency 
(FEMA)  regional  offices  that  had  exercised  with  NORTHCOM  in  three  of 
the  last  six  large-scale  exercises.  We  also  interviewed  officials  from  the 


1  For  purposes  of  this  report,  large-scale  exercises  are  those  involving  multiple  agencies  and 
organizations,  venues,  and  events,  such  as  NORTHCOM’s  Vigilant  Shield  and  Ardent  Sentry 
exercises.  NORTHCOM  generally  conducts  two  large-scale  exercises  annually. 
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Office  of  the  Assistant  Secretary  of  Defense  for  Homeland  Security  and 
America’s  Security  Affairs,  Joint  Staff,  and  NORTHCOM  with  knowledge 
of  and  experience  with  NORTHCOM’s  training  and  exercise  program. 

To  determine  the  extent  to  which  NORTHCOM  is  using  lessons  learned 
during  exercises  to  improve  mission  preparedness,  we  reviewed  DOD, 
NORTHCOM,  and  DHS  National  Exercise  Program  (NEP)  guidance  for 
recording,  tracking,  and  managing  lessons  learned  and  assessed 
NORTHCOM’s  management  of  exercise  observations  and  issues  identified 
in  several  of  NORTHCOM’s  large-scale  exercises  since  Hurricane  Katrina 
in  2005.  We  interviewed  NORTHCOM,  Joint  Staff,  and  FEMA  officials 
regarding  the  various  lessons  learned  management  systems,  and  how 
interagency  and  state  access  to  these  systems  can  be  accomplished.  We 
also  spoke  with  an  official  in  the  General  Services  Administration 
regarding  the  types  of  federal  personal  identification  verification  cards 
used  by  DOD  and  other  federal  departments  and  agencies  to  access 
government  computer  systems.  In  reviewing  the  management  of 
NORTHCOM’s  lessons  learned  program  we  identified  and  reviewed  all 
unclassified  exercise  observations  from  its  last  six  large-scale  exercises 
that  had  been  activated  in  NORTHCOM’s  area  of  the  Joint  Lessons 
Learned  Information  System  (JLLIS).  Our  review  of  the  records  in  JLLIS 
entailed  determining  each  record’s  status  (open  or  closed),  its  type  (issue 
or  lesson  learned),  and  each  record’s  disposition  after  NORTHCOM  staff 
have  acted  on  these  records  to  respond  to  the  issues  or  lessons  learned 
documented.  Based  on  our  review,  we  generally  placed  these  records  into 
one  of  several  categories:  open;  closed,  nonconcur;  issue  closed  with 
reobservation;  issue  closed  with  no  reobservation;  and  lesson  learned.  In 
addition  we  reviewed  several  records  that  had  been  merged  with  other 
original  records  because  each  related  to  the  same  issue;  however,  the 
original  record  for  that  issue  was  not  part  of  our  universe.  Therefore, 
without  reviewing  the  lead  record  the  merged  records  lacked  sufficient 
information  regarding  their  disposition  and  that  condition  became  another 
category. 

Finally,  to  determine  the  extent  to  which  NORTHCOM  is  integrating  its 
training  and  exercises  with  the  NEP  we  reviewed  DOD,  NORTHCOM,  and 
Department  of  Homeland  Security  guidance  to  identify  any  differences  in 
exercise  planning  and  documentation  between  DOD’s  guidance  and  that 
for  the  NEP.  We  used  that  analysis  to  determine  under  what  conditions 
NORTHCOM  should  apply  standards  related  to  the  NEP,  and  how  DOD 
and  its  subordinate  commands  should  participate  in  the  NEP  tier  I  or  II 
exercises.  We  reviewed  NORTHCOM  documentation  from  two  major 
national  exercises  conducted  during  fiscal  year  2008  to  determine  the 
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extent  to  which  NORTHCOM  employed  the  guidance  from  the  Homeland 
Security  Exercise  Evaluation  Program.  We  determined  that  national 
exercises  that  are  operations-based  exercises  in  that  they  involved  the 
deployment  of  personnel  would  be  the  best  candidates  for  evaluating 
NORTHCOM’s  participation  in  such  exercises.  We  also  interviewed  state 
emergency  management  and  National  Guard  officials  from  six  states  that 
have  exercised  with  NORTHCOM  since  2005,  to  understand  the  extent  to 
which  NORTHCOM  is  integrating  its  exercise  planning  and  conduct  with 
the  interagency  as  well  as  various  state  governments. 

In  addressing  our  objectives,  we  reviewed  plans  and  related  documents, 
obtained  information,  and  interviewed  officials  at  the  following  locations: 

•  NORTHCOM  Headquarters,  Peterson  Air  Force  Base,  Colorado 
Springs,  Colorado 

•  Joint  Forces  Command,  Joint  Warfighting  Center,  Suffolk,  Virginia 

•  The  Office  of  the  Secretary  of  Defense,  Washington,  D.C. 

•  The  Joint  Staff,  Washington,  D.C. 

•  U.S.  Army  North,  Fort  Sam  Houston,  San  Antonio,  Texas 

•  National  Guard  Bureau,  Arlington,  Virginia 

•  Department  of  Homeland  Security,  Washington,  D.C. 

•  U.S.  Coast  Guard,  Atlantic  Area,  Portsmouth,  VA 

•  FEMA’s  National  Preparedness  Directorate,  Washington,  D.C. 

•  FEMA  Region  1,  Boston,  Massachusetts 

•  FEMA  Region  9,  Oakland,  California 

•  FEMA  Region  10,  Bothell,  Washington 

•  General  Services  Administration,  Washington,  D.C. 

•  Arizona  Division  of  Emergency  Management,  Phoenix,  Arizona 

•  Arizona  National  Guard,  Joint  Force  Headquarters,  Phoenix,  Arizona 

•  California  Emergency  Management  Agency,  Sacramento,  California 

•  California  National  Guard,  Joint  Force  Headquarters,  Sacramento, 
California 

•  Michigan  State  Police,  Emergency  Management  and  Homeland 
Security  Division,  Lansing,  Michigan 

•  Michigan  National  Guard,  Lansing,  Michigan 

•  Nevada  State  Division  of  Emergency  Management,  Carson  City, 

Nevada 

•  Nevada  National  Guard,  Joint  Force  Headquarters,  Carson  City, 

Nevada 

•  Rhode  Island  Emergency  Management  Agency,  Cranston,  Rhode  Island 

•  Rhode  Island  National  Guard,  Joint  Force  Headquarters,  Cranston, 
Rhode  Island 

•  Oregon  Military  Department,  Office  of  Emergency  Management, 

Salem,  Oregon 
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•  Oregon  Military  Department,  National  Guard  Joint  Force 
Headquarters,  Salem,  Oregon 

•  Washington  Military  Department,  Emergency  Management  Division, 
Camp  Murray,  Washington 

•  Washington  National  Guard,  Joint  Force  Headquarters,  Camp  Murray, 
Washington 

We  conducted  our  review  from  June  2008  to  September  2009  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 
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We  reviewed  the  time  lines  and  milestones  for  developing  exercises  found 
in  the  Department  of  Defense’s  Joint  Training  System  and  U.S.  Northern 
Command’s  (NORTHCOM)  implementing  guidance  and  compared  them 
with  the  Department  of  Homeland  Security’s  Homeland  Security  Exercise 
and  Evaluation  Program  guidance  to  determine  the  similarities  and 
differences  between  them.  We  used  the  guidance  associated  with 
operation-based  exercises  rather  than  discussion-based  exercises  to 
present  the  full  spectrum  of  Homeland  Security  Exercise  and  Evaluation 
Program  processes  and  planning  events. 


Table  8:  Comparison  of  DOD  Joint  Training  System  and  Homeland  Security  Exercise  and  Evaluation  Program 


DOD  Joint  Training  System  and  NORTHCOM  Exercise 

Program  Homeland  Security  Exercise  and  Evaluation  Program  Guidance 

Joint  Event  Life  Cycle  Homeland  Security  Exercise  and  Evaluation  Program  Exercise  Cycle 


NORTHCOM  uses  the  Joint  Events  Life  Cycle  to  design,  plan, 

and  execute  exercises. 

•  Consists  of  5  stages:  design,  planning,  preparation, 
execution,  and  evaluation. 

•  Is  a  flexible  guide  that  can  be  modified  to  apply  to  various 
types  of  events 

•  Can  span  a  period  of  1 2  to  1 8  months  for  exercise 
development. 


Homeland  Security  Exercise  and  Evaluation  Program  policy  and 
guidance  is  the  doctrine  for  the  development  of  National  Exercise 
Program  exercises. 

•  Employs  5  phases,  which  are  collectively  known  as  the  exercise 
cycle:  foundation,  design  and  development,  conduct,  evaluation, 
and  improvement  planning. 

•  Planning  time  line  for  the  national  level  exercise  will  begin 
approximately  15  months  prior  to  exercise  conduct. 


Concept  Working  Group  Concept  and  Objective  Meeting 


Time  frame:  1 2  to  15  months  prior  to  start  of  exercise. 

Purpose: 

•  Commences  start  of  design  stage. 

•  Conduct  initial  planning  to  develop  the  candidate 
scenario  setting,  determine  funding  constraints,  and 
define  training  requirements. 

•  Result  in  the  validation  of  the  exercise  purpose,  exercise 
objectives,  identification  of  the  training  audience, 
intelligence  concepts,  draft  exercise  event  flow,  planning 
milestone,  and  draft  initial  planning  conference  agenda. 


Time  frame:  Minimum  6  months  prior  to  start  of  exercise,  lasts  2  to  4 

hours. 

Purpose: 

•  Marks  the  start  of  the  planning  process  and  can  be  conducted  in 
conjunction  with  the  initial  planning  conference. 

•  Identify  the  type,  scope,  objectives,  and  purpose  of  the  exercise, 
as  well  as  help  planners  identify,  among  other  things,  exercise 
planning  team  members. 

•  Reach  agreement  regarding  exercise  type,  scenario,  capabilities, 
tasks,  objectives,  the  target  exercise  time  frame,  and  the  date  and 
time  of  the  next  planning  conference. 


Page  71 


GAO-09-849  Homeland  Defense 


Appendix  II:  Comparison  of  DOD’s  Joint 
Training  System  with  DHS’s  Homeland 
Security  Exercise  and  Evaluation  Program 


DOD  Joint  Training  System  and  NORTHCOM  Exercise 

Program  Homeland  Security  Exercise  and  Evaluation  Program  Guidance 

Initial  Planning  Conference  Initial  Planning  Conference 

Time  frame:  10  tol  2  months  prior  to  start  of  exercise,  lasts  3  Time  frame:  Minimum  6  months  prior  to  start  of  exercise,  lasts  3  to  6 

to  4  days.  hours. 

Purpose:  Purpose: 

•  First  major  planning  conference  in  the  Joint  Events  Life  •  Begins  planning  process. 

Cycle  process.  .  Determine  exercise  scope  by  gathering  input  from  the  exercise 

•  Identify  levels  of  participation,  develop  deployment  and  planning  team,  design  requirements  and  conditions  (such  as 

logistic  requirements,  validate  the  exercise  milestones,  assumptions  and  artificialities),  objectives,  extent  of  play,  and 

develop  outline  of  the  exercise  plan  and  other  planning  scenario  variables  (such  as  time,  location,  hazard  selection), 

documents,  determine  modeling  and  simulation  .  Used  to  develop  exercise  documentation  by  obtaining  the 

requirements,  and  develop  the  mid-planning  conference  planning  team’s  input  on  exercise  location,  schedule,  duration, 
agenda.  and  other  relevant  details. 

Mid-Planning  Conference  Mid-Term  Planning  Conference 

Time  frame:  7-8  months  prior  to  start  of  exercise,  lasts  Time  frame:  Minimum  3  months  prior  to  start  of  exercise,  lasts 

approximately  3  days.  approximately  1  day. 

Purpose:  Purpose: 

•  Develop  a  draft  exercise  plan,  scenario,  modeling  and  •  Discuss  and  determine  exercise  organization  and  staffing 

simulation  architecture,  after-action  review  concept  of  concepts,  scenario  and  time  line  development,  scheduling, 

operations,  detailed  augmentation  requirements.  logistics,  and  administrative  requirements. 

•  Update  milestones  and  identify  funding  issues.  In  •  Review  draft  documentation  (such  as  scenario,  exercise  plan, 

addition,  workgroups  should  have  master  scenario  controller  and  evaluator  handbook,  master  scenario  events  list), 

events  lists  identified  with  an  understanding  of  who  is  .  Conduct  a  walkthrough  of  the  proposed  exercise  site  at  the  end  of 
responsible  for  completing  one.  the  conference 

The  Master  Scenario  Events  List  Development  Conference  The  Master  Scenario  Events  List  Conference 


Time  frame:  Conducted  before,  after,  or  in  conjunction  with 
the  mid-planning  conference. 

Purpose: 

•  Forum  for  the  initial  development  of  detailed  scenario 
events  and  the  master  scenario  events  list — a  detailed 
narrative  description  of  the  exercise  scenario  in 
chronological  sequence. 


Time  frame:  Conducted  separately  or  combined  with  mid-planning 
conference  or  final  planning  conference;  length  varies  from  4  to  8 
hours. 

Purpose: 

•  Focuses  on  developing  the  master  scenario  events  list — a 
chronological  list  that  supplements  the  exercise  scenario  with 
event  synopses;  expected  participant  responses;  capabilities, 
tasks,  and  objectives  to  be  addressed;  and  responsible 
personnel. 
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DOD  Joint  Training  System  and  NORTHCOM  Exercise 
Program 

Final  Planning  Conference 

Time  frame:  3  months  prior  to  start  of  exercise. 

Purpose: 

•  Provide  a  last  check  of  the  status  of  the  planning  and 
preparation  for  exercise  and  to  finalize,  publish,  and 
distribute  key  exercise  plans  and  documents  (such  as 
exercise  plan,  supporting  plans,  scenario,  after-action 
review  collection  management  plan,  exercise  manning 
status,  control  plan,  exercise  battle  rhythm,  and  modeling 
and  simulation  status). 

•  Finalize  the  construct  of  the  exercise  with  conference 
representatives. 

Master  Scenario  Events  List  Synchronization  Conference 

Time  frame:  Field  shortly  after  the  final  planning  conference 
and  at  least  3  weeks  prior  to  start  of  exercise. 

Purpose:  All  events  and  implementers  are  reviewed  for 
timing,  realism,  and  completeness. 

Tabletop  Exercises  and  Senior  Leadership  Seminars 

Time  frame:  Prior  to  start  of  exercise. 

Purpose: 

Tabletop  exercises  are  discussion-based  exercises  involving 
key  personnel  discussing  hypothetical  scenarios. 

•  Facilitating  understanding  of  concepts  and  processes, 
and  identifying  strengths  and  shortfalls. 

•  Provide  an  opportunity  to  address  new  or  side  issues,  or 
work  out  processes  prior  to  the  start  of  the  exercise. 

Senior  leadership  seminars  provide  senior  leaders  (flag 
officer,  general  officer,  and  senior  executive  service  level)  the 
opportunity  to  discuss  lessons  learned  from  previous 
exercises  and  the  upcoming  exercise. 

Event  Tests 


Homeland  Security  Exercise  and  Evaluation  Program  Guidance 

Final  Planning  Conference 

Time  frame:  Minimum  6  weeks  prior  to  start  of  exercise,  lasts 

approximately  1  day. 

Purpose: 

•  Final  forum  for  reviewing  exercise  processes  and  procedures. 

•  No  major  changes  to  the  design  or  scope  of  the  exercise,  or  its 
supporting  documentation,  should  take  place. 

•  Ensures  that  all  logistical  requirements  have  been  met,  all 
outstanding  issues  have  been  identified  and  resolved,  and  all 
exercise  products  are  ready  for  printing. 


See  master  scenario  events  list  conference  above. 


Tabletop  Exercises  and  Seminars 

Time  frame:  Not  specified. 

Purpose: 

Tabletop  exercises  involve  key  personnel  discussing  hypothetical 

scenarios  in  an  informal  setting. 

•  Used  to  assess  plans,  policies,  and  procedures  or  to  assess  the 
systems  needed  to  guide  the  prevention  of,  response  to,  and 
recovery  from  a  defined  incident. 

Seminars  are  informal  discussions,  unconstrained  by  real-time 

portrayal  of  events  and  led  by  a  presenter. 

•  Orient  participants  to,  or  provide  an  overview  of,  authorities, 
strategies,  plans,  policies,  procedures,  protocols,  response 
resources,  and/or  concepts  and  ideas. 

Exercise  Setup 


Time  frame.  Prior  to  start  of  exercise.  Timeframe:  Prior  to  start  of  exercise. 

Purpose.  Conducted  to  ensure  the  modeling  and  simulation  .  Exercise  planning  team  assigned  to  setup  should  visit  the 
architecture  is  sufficient  to  meet  exercise  requirements.  exercise  site  at  least  1  day  prior  to  the  event  to  arrange  the  room 

and  test  audio/video  equipment. 

•  On  the  day  of  the  exercise,  planning  team  members  should  arrive 
several  hours  before  start  of  exercise  to  handle  any  remaining 
logistical  or  administrative  items  pertaining  to  setup,  and  to 
arrange  for  registration. 
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DOD  Joint  Training  System  and  NORTHCOM  Exercise 
Program 

Homeland  Security  Exercise  and  Evaluation  Program  Guidance 

Communications  Exercise 

See  Exercise  Setup  above 

Time  frame:  Conducted  approximately  2  weeks  prior  to  start 
of  exercise. 

Purpose:  Confirm  all  exercise  NORAD  and  NORTHCOM  staff 
and  external  battle  staff  members  have  access  to  and  are 
proficient  with  all  required  communications  systems  and 
collaborative  tools. 

Exercise  Execution 

Exercise  Conduct 

Time  frame:  Start  of  exercise. 

•  Executed  per  the  exercise  plan. 

•  Depending  on  the  exercise,  it  may  only  involve  the 
execution  of  a  command  post  exercise  or  be  greater  in 
scope  and  involve  the  simultaneous  and/or  concurrent 
execution  of  other  training  events. 

Time  frame:  Start  of  exercise. 

•  Includes  setup  and  wrap-up  activities. 

•  For  an  operations-based  exercise,  conduct  encompasses  all 
operations  occurring  between  the  designated  start  of  the  exercise 
and  the  end  of  the  exercise. 

Facilitated  After-Action  Review 

Hot  Wash 

Time  frame:  After  the  end  of  the  exercise. 

Purpose:  A  facilitated  discussion  held  at  the  end  of  the 
exercise  convened  for  the  benefit  of  the  training  audience  to 
examine  actions  and  results  during  a  training  event. 

Time  frame:  Maximum  2  hours  after  end  of  exercise. 

Purpose: 

•  Provides  opportunity  to  ascertain  the  level  of  satisfaction  with  the 
exercise,  identify  issues  or  concerns,  and  propose  areas  for 
improvement. 

•  Players  complete  and  submit  their  Participant  Feedback  Forms 
during  the  hot  wash. 

•  All  evaluators  take  notes  during  play  and  hot  washes  for  later 
compilation  with  other  observations  from  their  functional  areas. 

After-Action  Review 

Controller  and  Evaluator  Debrief 

Time  frame:  May  occur  during  and  after  the  exercise. 

Purpose:  An  informal  process  designed  by  a  commander  or 
director  to  provide  commanders  direct  feedback  on  the 
accomplishment  of  selected  joint  mission-essential  tasks, 
conditions,  and  standards  stated  in  terms  of  training 
objectives  for  the  commander  to  evaluate  training  proficiency. 

Time  frame:  Maximum  1  day  after  end  of  exercise. 

Purpose:  Exercise  planning  team  leader  facilitates  this  debrief,  which 
provides  each  controller  and  evaluator  with  an  opportunity  to  provide 
an  overview  of  the  functional  area  he  or  she  observed  and  to  discuss 
both  strengths  and  areas  for  improvement. 

See  facilitated  after-action  review. 

After-Action  Conference 

Time  frame:  5  weeks  after  end  of  exercise,  lasts  approximately  1  day. 
Purpose: 

•  Present,  discuss,  and  refine  the  draft  after-action  report,  and  to 
develop  an  improvement  plan. 

•  Solicit  feedback  and  make  necessary  changes  to  after-action 
report. 

Source:  GAO  analysis  of  DOD  and  DHS  documentation. 
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ASSISTANT  SECRETARY  OF  DEFENSE 
2600  DEFENSE  PENTAGON 
WASHINGTON,  D.C.  20301-2600 


AUG  1  1  2009 


Ms.  Davi  M.  D'Agostino 
Director,  Defense  Capabilities  and  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington.  DC  20548 

Dear  Ms.  D’Agostino: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report,  GAO- 
09-849,  “HOMELAND  DEFENSE:  U.S.  Northern  Command  Has  a  Strong  Exercise 
Program,  but  involvement  of  Interagency  Partners  and  States  Can  Be  Improved,”  dated 
July  17,  2009  (GAO  Code  351178).  DoD  concurs  with  lour  recommendations,  and 
partially  concurs  with  three  recommendations.  Our  response  to  the  recommendations  is 
enclosed. 

Our  point  of  contact  for  this  action  is  Mr.  Johnnie  Wauchop,  Office  of  the 
Assistant  Secretary  of  Defense  for  Homeland  Defense  and  Americas’  Security  Affairs 


Enclosure: 
As  stated 
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GAO  DRAFT  REPORT  -  DATED  JULY  17,  2009 
GAO  09-849(351178) 

"HOMELAND  DEFENSE:  U.S.  Northern  Command  Has  a  Strong  Exercise  Program,  but 
Involvement  of  Interagency  Partners  and  States  Can  Be  Improved" 


DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  RECOMMENDATIONS 

RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  direct  the  U.S. 
Northern  Command's  (USNORTHCOM)  Commander  to  develop  guidance  with  specific  criteria 
for  post-exercise  documentation,  particularly  the  Exercise  Summary  Report  as  the  official 
exercise  record,  including  the  content  and  format  to  be  included  in  such  reports  that  will  allow  the 
results  and  lessons  learned  of  exercises  to  be  easily  reviewed  and  compared. 

POD  RESPONSE:  Partially  Concur.  DoD  agrees  that  such  information,  when  appropriate  and 
approved  for  release,  should  be  provided  in  a  standardized  format  that  can  be  easily  accessed  and 
understood  by  authorized  organizations  which  may  benefit  from  the  knowledge  acquired  through 
our  exercise  activities.  Any  solutions  must  be  able  to  accommodate  constraints  regarding 
classified  information. 


RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense,  in  consultation 
with  the  Chairman  of  the  Joint  Chiefs  of  Staff,  the  Commander,  USNORTHCOM,  and  other 
relevant  Combatant  Commanders,  coordinate  with  the  Department  of  Homeland  Security  (DHS) 
and  Federal  Emergency  Management  Agency  (FEMA)  to  develop  guidance  and  procedures  for 
consistently  involving  state  officials  in  planning,  executing,  and  assessing  exercises  that 
incorporate  relevant  state-specific  information. 

DOD  RESPONSE.  Partially  Concur.  DoD  concurs  that  better  coordination  for  interfacing  with 
State  officials  can  be  achieved.  However,  USNORTHCOM  must  rely  on  State  and  local  exercise 
planning  representatives  to  accurately  and  thoroughly  represent  their  State  Emergency 
Management  structure  and  their  training  objectives  during  the  12-month  planning  process  leading 
up  to  each  exercise.  To  this  end,  USNORTHCOM  continues  to  expand  its  efforts  to  work 
through  its  Defense  Coordinating  Officers/Elements  (DCOs/DCEs),  its  existing  relationships  with 
State  National  Guard  forces,  and  with  FEMA  regional  headquarters  partners  to  ensure  that  States 
are  also  able  to  benefit  from  participating  in  DoD-sponsored  homeland  security  related  exercises. 

USNORTHCOM,  throughout  the  history  of  its  exercise  program,  has  continuously  engaged  and 
encouraged  the  participation  of  State  emergency  management  officials/struclures  in  order  to 
maximize  available  training  opportunities  for  all  stakeholders.  Additionally,  as  noted  in  the  GAO 
report,  there  are  ample  references  to  the  amount  of  State  “play"  and  the  benefits  that  States  have 
derived  trom  USNORTHCOM-sponsored  exercises.  Ultimately,  however,  it  must  be 
acknowledged  that  the  primary  training  audience  in  any  USNORTHCOM-sponsored  exercise  is 
USNORTHCOM. 

Finally,  the  GAOs  proposed  recommendation  has  applicability  to  other  US  Government 
interagency  partners,  as  well  as  DoD.  As  such,  the  issue  should  be  addressed  by  the  Exercise  and 
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Evaluation  Sub-Interagency  Planning  Committee  as  a  revision  to  the  National  Exercise  Program 
(NEP)  Implementation  Plan. 


RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
USNORTHCOM’s  Commander  to  develop  a  training  plan  for  USNORTHCOM  headquarters 
staff  on  State  emergency  management  structures  and  relevant  issues  related  to  working  with 
civilian  State  and  local  emergency  management  officials. 

POD  RESPONSE:  Concur.  Defense  Coordinating  Officers,  who  report  to  USNORTHCOM  via 
U.S.  Army  North  (USARNORTH),  are  well-versed  in  their  respective  States’  procedures. 
Additionally,  the  headquarters  staff  training  specified  in  the  recommendation,  which  has  been  in 
existence  since  2002,  is  required  for  all  newly  assigned  USNORTHCOM  staff  members.  The 
training  includes  a  full  discussion  of  how  States  request  and  receive  Defense  Support  of  Civil 
Authorities  (DSCA).  Further,  USNORTHCOM  sponsors  three  versions  of  its  formal  DSCA 
seminar  which  are  described  as  follows: 

•  A  three  phase  DSCA  course  that  includes  eight  hours  of  online  instruction,  a  one- week 
classroom  seminar,  and  bi-monthly  post-training  e-mail  updates  on  DSCA  news  items. 

•  A  DSCA  Executive  Seminar  offered  four  times  per  year  and  tailored  to  senior  official 
training  audiences  in  DoD  (including  the  National  Guard),  interagency  organizations. 

Stale  emergency  management  agencies,  local  governments,  and  non-governmental 
organizations. 

•  A  soon-to-be  available  DSCA  Senior  Executive  Seminar,  a  focused  half-day  session 
intended  for  Governors,  State  Adjutants  General,  and  State  Directors  of  Emergency 
Management. 


RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
USNORTHCOM’s  Commander  to  establish  and  publicize  valid  and  easily  accessible  procedures 
for  non-DOD  exercise  participants  to  submit  observations  relevant  to  USNORTHCOM,  such  as 
placing  a  template  on  USNORTHCOM’s  publicly  accessible  website  or  the  DHS’s  Homeland 
Security  Information  Network,  so  that  USNORTHCOM  officials  have  a  clear,  secure  avenue  to 
obtain  observations  and  assess  potential  lessons  that  originate  with  its  exercise  partners. 

DOD  RESPONSE:  Concur.  Collecting  relevant  exercise  and  training  information  from  all 
perspectives  will  undoubtedly  provide  additional  opportunities  to  improve  USNORTHCOM’s 
ability  to  accomplish  its  mission  essential  tasks. 


RECOMMENDATION  5:  The  GAO  recommends  that  the  Secretary  of  Defense  direct  the 
Chairman  of  the  Joint  Chiefs  of  Staff  in  consultation  and  coordination  with  DHS,  to  either 
resolve  information  assurance  issues  so  that  the  combatant  commands,  including 
USNORTHCOM,  can  post  Exercise  Summary  Reports  with  lessons  learned  an<T observations 
from  the  National  Exercise  Program  (NEP)  exercises  on  DHS’s  Lessons  Learned  Information 
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Sharing  system  to  make  them  easily  accessible  to  interagency  partners  and  states  or  establish  an 
alternative  method  to  systematically  collect  and  share  lessons  learned. 

POD  RESPONSE:  Concur.  DoD  agrees  that  such  information,  when  appropriate  and  approved 
for  release,  should  be  disseminated  as  widely  as  possible  to  those  homeland  security  related 
entities  which  may  benefit  from  the  knowledge  acquired  through  our  exercise  activities. 
Concurrence,  however,  should  not  be  misconstrued  as  support  for  altering  or  departing  from  the 
Joint  Training  System  and  its  components,  specifically  the  Joint  Lessons  Learned  information 
System.  Additionally,  DoD  cannot  mandate  that  DHS  alter  its  Lessons  Learned  Information 
System  nor  require  DHS  to  commit  resources  to  do  so.  There  are  procedures  in  place  that  allow 
specifically  cleared  individuals  from  outside  DoD  access  to  information  contained  in  the  Exercise 
Summary  Reports.  These  manual  procedures  ensure  that  only  appropriate  and  authorized 
information  is  offered  outside  the  Department. 


RECOMMENDATION  6:  The  GAO  recommends  that  the  Secretary  of  Defense  direct  the 
Chairman  of  the  Joint  Chiefs  of  Staff  to  revise  the  joint  lessons  learned  operating  instruction  to 
include  procedures  to  ensure  that  appropriate  corrective  actions  are  implemented  and  verified  in  a 
subsequent  exercise  or  operation  before  being  closed  and  that  the  reasons  for  closure  are 
documented.  Possible  procedures  might  be  adding  a  verification  checkbox  on  the  Joint  Lessons 
Learned  Information  System's  (JLLIS)  issue  management  page  or  requiring  that  the  directorates 
and  subordinate  commands  within  the  combatant  commands  provide  a  status  report  when  a 
correction  is  implemented  and  re-observed  or  closed  for  reasons  other  than  re-observaticn. 

DOD  RESPONSE:  Concur.  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCS1) 

3 1 50.25D  provides  policy  and  guidance  for  the  Joint  Lessons  Learned  Program  and  depicts  the 
“revalidation”  of  issues  to  ensure  lessons  have  been  learned  (Figure  A-2,  Joint  Lessons  Learned 
Program  Process  Overview).  This  discussion  section  could  be  expanded  to  provide  more 
guidance  and  the  JLLIS  could  be  updated  to  provide  a  technological  solution  to  address  rhe  issue 
once  the  process  and  procedures  are  in  place. 

Regarding  internal  tracking,  JLLIS  provides  sufficient  capability  for  directors  and  subordinate 
commands  to  effectively  manage  internal  issues.  JLLIS  Tier  I  administrators  can  make  local 
modifications  to  facilitate  this  process.  The  process  of  verifying  corrective  action  and  closing 
issues  will  become  more  effective  with  the  suggested  modification,  continued  education,  and 
further  maturation  of  JLLIS  and  the  Joint  Lessons  Learned  Program. 


RECOMMENDATION  7:  The  GAO  recommends  that  the  Secretary  of  Defense  revise  the 
instruction  on  DOD  participation  in  the  NEP  and/or  direct  the  Chairman  of  the  Joint  Chiefs  of 
Staff  to  revise  the  operating  instruction  regarding  DOD  participation  in  the  NEP  to  provide  the 
general  conditions  under  which  the  combatant  commands  are  expected  to  follow  the  Homeland 
Security  Exercise  and  Evaluation  Program  planning  and  documentation  requirements  or  the 
DOD’s  Joint  Training  System  should  be  modified  for  those  civil  support  exercises. 

DOD  RESPONSE:  Partially  Concur.  DoD  recognizes  the  importance  of  ensuring  the 
Department  is  sufficiently  able  to  interact  with  our  interagency  partners  at  every  level  regarding 
the  planning  and  execution  of  Homeland  Security-related  exercises.  To  this  end,  DoD  has  fully 
supported  and,  when  appropriate,  has  directly  assisted  in  the  development  of  the  NEP  and  its 
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underlying  doctrine,  methodologies,  and  technologies.  Evidence  of  these  efforts  are  readily  found 
m  the  GAO  report  where  the  similarities  between  DoD  exercise  methodology  and  National 
Exercise  Program  doctrine  are  obvious. 

The  NEP  Implementation  Plan  (I-Plan)  clearly  delineates  how  DoD  participates  in  the  NEP. 
Additionally,  largely  due  to  DoD’s  insistence,  the  1-Plan  contains  language  in  paragraph  6.2 
establishing  processes  to  resolve  doctrinal  differences  during  the  initial  stages  of  NEP  exercise 
planning.  Further,  the  NEP  I-Plan  establishes  the  administration,  scope,  and  hierarchy  of 
homeland  security-related  exercises  involving  multiple  interagency  entities.  Combined  with  its 
associated  NEP  five-year  schedule  which  includes  USNORTHCO.M  homeland  security-related 
exercises,  the  1-Plan  and  the  five-year  schedule  appropriately  address  the  GAO's 
recommendation. 

It  should  be  emphasized,  however,  that  the  Secretary  of  Defense  retains  sole  authority  for  training 
and  equipping  the  Department  of  Defense.  The  Joint  Training  System,  as  it  is  currently 
administered,  constitutes  the  Secretary’s  guidance  regarding  this  matter.  This  position  is  further 
reinforced  in  paragraph  4  of  the  NEP  Implementation  plan,  which  states  “The  NEP  is  intended  to 
provide  a  framework  for  prioritizing  and  focusing  Federal  exercise  activities,  without  replacing 
any  individual  department  or  agency  exercise  program”. 
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